Collapsed Gibbs Sampling Algorithm

Context:
- It can be applied by a Collapsed Gibbs Sampling System (to solve a collapsed Gibbs sampling task).
- …
Counter-Example(s):
- Blocked Gibbs Sampling Algorithm.
See: Jonathan Chang's LDA R Package.

References

http://en.wikipedia.org/wiki/Gibbs_sampling#Variations_and_extensions
- A 'blocked Gibbs sampler groups two or more variables together and samples from their joint distribution conditioned on all other variables, rather than sampling from each one individually. For example, in a hidden Markov model, a blocked Gibbs sampler might sample from all the latent variables making up the Markov chain in one go, using the forward-backward algorithm.
  A collapsed Gibbs sampler integrates out (marginalizes over) one or more variables when sampling for some other variable. For example, imagine that a model consists of three variables A, B, and C. A simple Gibbs sampler would sample from p(A|B,C), then p(B|A,C), then p(C|A,B). A collapsed Gibbs sampler might replace the sampling step for A with a sample taken from the marginal distribution p(A|C), with variable B integrated out in this case. Alternatively, variable B could be collapsed out entirely, alternately sampling from p(A|C) and p(C|A) and not sampling over B at all. The distribution over a variable A that arises when collapsing a parent variable B is called a compound distribution; sampling from this distribution is generally tractable when B is the conjugate prior for A, particularly when A and B are members of the exponential family. For more information, see the article on compound distributions or Liu (1994).

(Liu, 1994) ⇒ Jun S. Liu. (1994). “The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem.” In: Journal of the American Statistical Association, 89(427). http://www.jstor.org/stable/2290921
- ABSTRACT: This article describes a method of "grouping" and "collapsing" in using the Gibbs sampler and proves from an operator theory viewpoint that the method is in general beneficial. The norms of the forward operators associated with the corresponding non-reversible Markov chains are used to discriminate among different simulation schemes. When applied to Bayesian missing data problems, the idea of collapsing suggests skipping the steps of sampling parameter(s) values in standard data augmentation. By doing this, we obtain a predictive update version of the Gibbs sampler. A procedure of calculating the posterior odds ratio via the collapsed Gibbs sampler when incomplete observations are involved is presented. As an illustration of possible applications, three examples, along with a Bayesian treatment for identifying common protein binding sites in unaligned DNA sequences, are provided.