Maximum a Posteriori Estimation Task
A Maximum a Posteriori Estimation Task is a point estimation task that requires a maximum a posteriori estimate.
- AKA: MAP Estimation.
- Context:
- It can be solved by a Maximum a Posteriori Estimation System (that implements a MAP estimation algorithm).
- It can be used to make a MAP Inference.
- Example(s):
- Decode an HMM (finding the sequence of tags which are most compatible/most likely to have been generated) by an HMM-based Tagging Model using MAP, i.e. [math]\displaystyle{ f(y)=argmax_{x∈Y}p(x)p(y|x) }[/math]
- Counter-Example(s):
- See: A Posteriori Estimate Task, Maximum Lixelihood Estimation, (Dempster et al., 1977), argmax, Bayesian Inference, Viterbi Decoding, MAP-based Classifier.
References
2015
- (Mohamed, 2015) ⇒ Shakir Mohamed (2015). “A Statistical View of Deep Learning (V): Generalisation and Regularisation.” In: Personal Blog, 10 May 2015
- QUOTE: The principle technique for addressing overfitting in deep learning is by regularisation — adding additional penalties to our training objective that prevents the model parameters from becoming large and from fitting to the idiosyncrasies of the training data. This transforms our estimation framework from maximum likelihood into a maximum penalised likelihood, or more commonly maximum a posteriori (MAP) estimation (or a shrinkage estimator). For a deep model with loss function L(θ) and parameters θ, we instead use the modified loss that includes a regularisation function R:
L(θ)=−∑nlogp(yn|xn,θ)+1λR(θ)
- QUOTE: The principle technique for addressing overfitting in deep learning is by regularisation — adding additional penalties to our training objective that prevents the model parameters from becoming large and from fitting to the idiosyncrasies of the training data. This transforms our estimation framework from maximum likelihood into a maximum penalised likelihood, or more commonly maximum a posteriori (MAP) estimation (or a shrinkage estimator). For a deep model with loss function L(θ) and parameters θ, we instead use the modified loss that includes a regularisation function R:
2011
- (Wikipedia, 2011) http://en.wikipedia.org/wiki/Maximum_a_posteriori
- In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is a mode of the posterior distribution. The MAP can be used to obtain a point estimate of an unobserved quantity on the basis of empirical data. It is closely related to Fisher's method of maximum likelihood (ML), but employs an augmented optimization objective which incorporates a prior distribution over the quantity one wants to estimate. MAP estimation can therefore be seen as a regularization of ML estimation.
Assume that we want to estimate an unobserved population parameter [math]\displaystyle{ \theta }[/math] on the basis of observations [math]\displaystyle{ x }[/math]. Let [math]\displaystyle{ f }[/math] be the sampling distribution of [math]\displaystyle{ x }[/math], so that [math]\displaystyle{ f(x|\theta) }[/math] is the probability of [math]\displaystyle{ x }[/math] when the underlying population parameter is [math]\displaystyle{ \theta }[/math]. Then the function [math]\displaystyle{ \theta \mapsto f(x | \theta) \! }[/math] is known as the likelihood function and the estimate [math]\displaystyle{ \displaystyle \hat{\theta}_{\mathrm{ML}}(x) = \mathop{\mbox{arg max }}_{\theta}\ f(x \vert \theta) \! }[/math] is the maximum likelihood estimate of [math]\displaystyle{ \theta }[/math]. Now assume that a prior distribution [math]\displaystyle{ g }[/math] over [math]\displaystyle{ \theta }[/math] exists. This allows us to treat [math]\displaystyle{ \theta }[/math] as a random variable as in Bayesian statistics. Then the posterior distribution of [math]\displaystyle{ \theta }[/math] is as follows: [math]\displaystyle{ \theta \mapsto f(\theta | x) = \frac{f(x | \theta) \, g(\theta)}{\displaystyle \int_{\theta' \in \Theta} f(x | \theta') \, g(\theta') \, d\theta'} \! }[/math] where [math]\displaystyle{ g }[/math] is density function of [math]\displaystyle{ \theta }[/math], [math]\displaystyle{ \Theta }[/math] is the domain of [math]\displaystyle{ g }[/math]. This is a straightforward application of Bayes' theorem. The method of maximum a posteriori estimation then estimates [math]\displaystyle{ \theta }[/math] as the mode of the posterior distribution of this random variable: [math]\displaystyle{ \displaystyle \hat{\theta}_{\mathrm{MAP}}(x) = \mathop{\mbox{arg max }}_{\theta} \ \frac{f(x | \theta) \, g(\theta)} {\displaystyle\int_{\Theta} f(x | \theta') \, g(\theta') \, d\theta'} = \mathop{\mbox{arg max }}_{\theta} \ f(x | \theta) \, g(\theta). \! }[/math]
The denominator of the posterior distribution (so-called partition function) does not depend on [math]\displaystyle{ \theta }[/math] and therefore plays no role in the optimization. Observe that the MAP estimate of [math]\displaystyle{ \theta }[/math] coincides with the ML estimate when the prior [math]\displaystyle{ g }[/math] is uniform (that is, a constant function). The MAP estimate is a limit of Bayes estimators under a sequence of 0-1 loss functions, but generally not a Bayes estimator per se, unless [math]\displaystyle{ \theta }[/math] is discrete.
- In Bayesian statistics, a maximum a posteriori probability (MAP) estimate is a mode of the posterior distribution. The MAP can be used to obtain a point estimate of an unobserved quantity on the basis of empirical data. It is closely related to Fisher's method of maximum likelihood (ML), but employs an augmented optimization objective which incorporates a prior distribution over the quantity one wants to estimate. MAP estimation can therefore be seen as a regularization of ML estimation.
2010
- (Lingpipe, 2010) ⇒ http://alias-i.com/lingpipe/demos/tutorial/logistic-regression/read-me.html
- MAP estimation with Gaussian, Laplace or Cauchy priors is known as parameter shrinkage.