Cross-Entropy Minimization Task

A Cross-Entropy Minimization Task is an numerical optimization task whose optimization function is a cross-entropy function.

• • …
• Counter-Example(s):
  • Accuracy Maximization.
• See: Kullback–Leibler Divergence, Cross-Entropy Method, Kullback–Leibler Divergence, Principle of Minimum Discrimination Information.

---

---

References

2017

• (Wikipedia, 2017) ⇒ https://en.wikipedia.org/wiki/Cross_entropy#Cross-entropy_minimization Retrieved:2017-6-7.
  • Cross-entropy minimization is frequently used in optimization and rare-event probability estimation; see the cross-entropy method.

    When comparing a distribution [math]\displaystyle{ q }[/math] against a fixed reference distribution [math]\displaystyle{ p }[/math] , cross entropy and KL divergence are identical up to an additive constant (since [math]\displaystyle{ p }[/math] is fixed): both take on their minimal values when [math]\displaystyle{ p = q }[/math] , which is [math]\displaystyle{ 0 }[/math] for KL divergence, and [math]\displaystyle{ \mathrm{H}(p) }[/math] for cross entropy. In the engineering literature, the principle of minimising KL Divergence (Kullback's “Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent.

    However, as discussed in the article Kullback–Leibler divergence, sometimes the distribution [math]\displaystyle{ q }[/math] is the fixed prior reference distribution, and the distribution [math]\displaystyle{ p }[/math] is optimised to be as close to [math]\displaystyle{ q }[/math] as possible, subject to some constraint. In this case the two minimisations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be [math]\displaystyle{ D_{\mathrm{KL}}(p \| q) }[/math] , rather than [math]\displaystyle{ H(p, q) }[/math] .

---