2005 VariationalMethodsForDPM
- (Blei & Jordan, 2005) ⇒ David M. Blei, and Michael I. Jordan. (2005), "Variational Methods for Dirichlet Process Mixtures.” In: Bayesian Analysis, 1.
Subject Headings: Variational Inference Algorithm, Dirichlet Process, Hierarchical Model. Dirichlet processes, hierarchical models, variational inference, image processing, Bayesian computation
Notes
- Reference for: MCMC Algorithm, …
Cited By
Quotes
Abstract
Dirichlet process (DP) mixture models are the cornerstone of non-parametric Bayesian statistics, and the development of Monte-Carlo Markov chain (MCMC) sampling methods for DP mixtures has enabled the application of non-parametric Bayesian methods to a variety of practical data analysis problems. However, MCMC sampling can be prohibitively slow, and it is important to explore alternatives. One class of alternatives is provided by variational methods, a class of deterministic algorithms that convert inference problems into optimization problems (Opper and Saad 2001; Wainwright and Jordan 2003). Thus far, variational methods have mainly been explored in the parametric setting, in particular within the formalism of the exponential family (Attias 2000; Ghahramani and Beal 2001; Blei et al. 2003). In this paper, we present a variational inference algorithm for DP mixtures. We present experiments that compare the algorithm to Gibbs sampling algorithms for DP mixtures of Gaussians and present an application to a large-scale image analysis problem.
1 Introduction
The methodology of Monte Carlo Markov chain (MCMC) sampling has energized Bayesian statistics for more than a decade, providing a systematic approach to the computation of likelihoods and posterior distributions, and permitting the deployment of Bayesian methods in a rapidly growing number of applied problems. However, while an unquestioned success story, MCMC is not an unqualified one -- MCMC methods can be slow to converge and their convergence can be difficult to diagnose. While further research on sampling is needed, it is also important to explore alternatives, particularly in the context of large-scale problems.
One such class of alternatives is provided by variational inference methods (Ghahramani and Beal 2001; Jordan et al. 1999; Opper and Saad 2001;Wainwright and Jordan 2003; Wiegerinck 2000). Like MCMC, variational inference methods have their roots in statistical physics, and, in contradistinction to MCMC methods, they are deterministic. The basic idea of variational inference is to formulate the computation of a marginal or conditional probability in terms of an optimization problem. This (generally intractable) problem is then "relaxed," yielding a simplified optimization problem that depends on a number of free parameters, known as variational parameters. Solving for the variational parameters gives an approximation to the marginal or conditional probabilities of interest.
Variational inference methods have been developed principally in the context of the exponential family, where the convexity properties of the natural parameter space and the cumulant function yield an elegant general variational formalism (Wainwright and Jordan 2003). For example, variational methods have been developed for parametric hierarchical Bayesian models based on general exponential family specifications (Ghahramani and Beal 2001). MCMC methods have seen much wider application. In particular, the development of MCMC algorithms for nonparametric models such as the Dirichlet process has led to increased interest in nonparametric Bayesian methods. In the current paper, we aim to close this gap by developing variational methods for Dirichlet process mixtures.
The Dirichlet process (DP), introduced in Ferguson (1973), is a measure on measures. The DP is parameterized by a base distribution G0 and a positive scaling parameter …
In this paper, we present a variational inference algorithm for DP mixtures based on the stick-breaking representation of the underlying DP. The algorithm involves two probability distributions -- the posterior distribution [math]\displaystyle{ p }[/math] and a variational distribution q.
References
,
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2005 VariationalMethodsForDPM | Michael I. Jordan David M. Blei | Variational Methods for Dirichlet Process Mixtures | Bayesian Analysis Task | http://www.cs.berkeley.edu/~jordan/papers/blei-jordan-ba.pdf | 2005 |