Negative Log-Likelihood

A Negative Log-Likelihood is a non-negative function based on the logarithm of the negation of a likelihood ratio.

• AKA: Negated Log-Likelihood.
• Context:
  • It can be calculated as [math]\displaystyle{ LLR = ln (- \frac{(P(I \vert D)/P(\sim I \vert D))}{P(I)/P(\sim I)}) }[/math] where [math]\displaystyle{ P(I\vert D) }[/math] and [math]\displaystyle{ P(\sim I\vert D) }[/math] are the frequencies of interactions observed in the given dataset (D) between annotated genes sharing benchmark associations (I) and not sharing associations (~I), respectively, while P(I) and P(~I) represent the prior expectations (the total frequencies of all benchmark genes sharing the same associations and not sharing associations, respectively).
  • …
• Counter-Example(s):
  • Log-Likelihood Ratio.
• See Discriminative Model, Generative Model, Negative Log-Likelihood Classification Loss, Classification Loss.

---

---

References

2011

• https://quantivity.wordpress.com/2011/05/23/why-minimize-negative-log-likelihood/
  • QUOTE:
    Why is minimizing the negative log likelihood equivalent to maximum likelihood estimation (MLE)?

    Or, equivalently, in Bayesian-speak:
    Why is minimizing the negative log likelihood equivalent to maximum a posteriori probability (MAP), given a uniform prior?

2009

• http://www8.tfe.umu.se/forskning/Control_Systems/Courses/System_Identification/chapter7.pdf
  • QUOTE: Negative log likelihood function: Suppose [math]\displaystyle{ Y_N = [y_{(1)}, ..., y_{(N)}] }[/math] is a Gaussian random vector such that [math]\displaystyle{ E{Y_N} = 0 }[/math] and [math]\displaystyle{ E {Y_N Y^T_N} = R_N(\theta) }[/math].
    • 1. Show that negative log likelihood function when [math]\displaystyle{ Y_N }[/math] is observed is [math]\displaystyle{ V(Y_N; \theta) = const + \frac{1}{2} \log \det R_N(\theta) + \frac{1}{2} Y^T_N R^{−1}_N (\theta)Y_N. }[/math]

2008

• (Lin et al., 2008) ⇒ Chih-Jen Lin, Ruby C. Weng, and S. Sathiya Keerthi. (2008). “Trust Region Newton Method for Logistic Regression.” In: The Journal of Machine Learning Research, 9.
  • QUOTE: The logistic regression model is useful for two-class classification. Given data [math]\displaystyle{ \mathbf{x} }[/math] and weights [math]\displaystyle{ (\mathbf{w},b) }[/math], it assumes the following probability model :[math]\displaystyle{ P(y=±1 | \mathbf{x},\mathbf{w}) = \frac{1}{1 + exp(-y(\mathbf{w}^T \mathbf{x} + b)}, }[/math] where [math]\displaystyle{ y }[/math] is the class label. If training instances are [math]\displaystyle{ x_i }[/math], [math]\displaystyle{ i=1,...,l }[/math] and labels are [math]\displaystyle{ y_i \in {1,-1}, }[/math] one estimates </math>(\mathbf{w}; b)</math> by minimizing the negative log-likelihood: :[math]\displaystyle{ \operatorname{min}_{\mathbf{x},b} \sum_{i=1}^{l} \log (1 + e^{-y_i(w^T x_i + b)}) }[/math]

---