Restricted Boltzmann Machine (RBM) Training System

A Restricted Boltzmann Machine (RBM) Training System is a Boltzmann Machine Training System that implements a Restricted Boltzmann Machine Training Algorithm to solve a Restricted Boltzmann Machine Training Task to produce a Restricted Boltzmann Machine ouput.

• AKA: RBM Training System.
• Example(s):
  • sklearn.neural_network.BernoulliRBM [1].
  • https://github.com/odie2630463/Restricted-Boltzmann-Machines-in-pytorch
  • …
• Counter-Example(s):
  • a Multi-layer Perceptron Classification System,
  • a Support Vector Machine Training System.
• See: Recurrent Neural Network, Generative Model, Stochastic Neural Network, Artificial Neural Network, Feature Learning, Autoencoder.

---

---

References

2018

• (sklearn,2018) ⇒ http://scikit-learn.org/stable/modules/neural_networks_unsupervised.html#rbm Retrieved:2018-03-11
  • QUOTE: Restricted Boltzmann machines (RBM) are unsupervised nonlinear feature learners based on a probabilistic model. The features extracted by an RBM or a hierarchy of RBMs often give good results when fed into a linear classifier such as a linear SVM or a perceptron.

    The model makes assumptions regarding the distribution of inputs. At the moment, scikit-learn only provides BernoulliRBM, which assumes the inputs are either binary values or values between 0 and 1, each encoding the probability that the specific feature would be turned on.

    The RBM tries to maximize the likelihood of the data using a particular graphical model. The parameter learning algorithm used (Stochastic Maximum Likelihood) prevents the representations from straying far from the input data, which makes them capture interesting regularities, but makes the model less useful for small datasets, and usually not useful for density estimation.

    The method gained popularity for initializing deep neural networks with the weights of independent RBMs. This method is known as unsupervised pre-training (...)

2017a

• (Hinton, 2017) ⇒ Hinton G. (2017). "Boltzmann Machines". In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA
  • QUOTE: A restricted Boltzmann machine (Smolensky 1986) consists of a layer of visible units and a layer of hidden units with no visible-visible or hidden-hidden connections. With these restrictions, the hidden units are conditionally independent given a visible vector, so unbiased samples from [math]\displaystyle{ \langle s_i s_j\rangle_{data} }[/math] can be obtained in one parallel step. To sample from [math]\displaystyle{ \langle s_is_j\rangle_{model} }[/math] still requires multiple iterations that alternate between updating all the hidden units in parallel and updating all of the visible units in parallel. However, learning still works well if [math]\displaystyle{ \langle s_is_j\rangle_{model} }[/math] is replaced by [math]\displaystyle{ \langle s_is_j\rangle_{reconstruction} }[/math] which is obtained as follows:

1. Starting with a data vector on the visible units, update all of the hidden units in parallel.
2. Update all of the visible units in parallel to get a “reconstruction.”
3. Update all of the hidden units again.

This efficient learning procedure approximates gradient descent in a quantity called “contrastive divergence” and works well in practice (Hinton 2002).

2017b

• (Wikipedia, 2017) ⇒ https://en.wikipedia.org/wiki/restricted_Boltzmann_machine Retrieved:2017-2-27.
  • A restricted Boltzmann machine (RBM) is a generative stochastic artificial neural network that can learn a probability distribution over its set of inputs.

    RBMs were initially invented under the name Harmonium by Paul Smolensky in 1986,, and rose to prominence after Geoffrey Hinton and collaborators invented fast learning algorithms for them in the mid-2000s. RBMs have found applications in dimensionality reduction, classification, collaborative filtering, feature learning and topic modelling. They can be trained in either supervised or unsupervised ways, depending on the task.

    As their name implies, RBMs are a variant of Boltzmann machines, with the restriction that their neurons must form a bipartite graph:

    a pair of nodes from each of the two groups of units (commonly referred to as the "visible" and "hidden" units respectively) may have a symmetric connection between them; and there are no connections between nodes within a group. By contrast, "unrestricted" Boltzmann machines may have connections between hidden units. This restriction allows for more efficient training algorithms than are available for the general class of Boltzmann machines, in particular the gradient-based contrastive divergence algorithm.

    Restricted Boltzmann machines can also be used in deep learning networks. In particular, deep belief networks can be formed by "stacking" RBMs and optionally fine-tuning the resulting deep network with gradient descent and backpropagation.

2017c

• (The Asimov Institute, 2017) ⇒ http://asimovinstitute.org/neural-network-zoo/
  • QUOTE: Restricted Boltzmann machines (RBM) are remarkably similar to BMs (surprise) and therefore also similar to HNs. The biggest difference between BMs and RBMs is that RBMs are a better usable because they are more restricted. They don’t trigger-happily connect every neuron to every other neuron but only connect every different group of neurons to every other group, so no input neurons are directly connected to other input neurons and no hidden to hidden connections are made either. RBMs can be trained like FFNNs with a twist: instead of passing data forward and then back-propagating, you forward pass the data and then backward pass the data (back to the first layer). After that you train with forward-and-back-propagation.
    

2014

• http://deeplearning4j.org/restrictedboltzmannmachine.html
  • QUOTE: To quote Geoff Hinton, a Google researcher and university professor, a Boltzmann machine is “a network of symmetrically connected, neuron-like units that make stochastic decisions about whether to be on or off.” (Stochastic means “randomly determined.”)

    A restricted Boltzmann machine “consists of a layer of visible units and a layer of hidden units with no visible-visible or hidden-hidden connections.” The “restricted” comes from limits imposed on how its nodes connect: intra-layer connections are not allowed, but each node of one layer connects to every node of the next, and that is called “symmetry.”

2012

• http://deeplearning.net/tutorial/rbm.html
  • Boltzmann Machines (BMs) are a particular form of log-linear Markov Random Field (MRF), i.e., for which the energy function is linear in its free parameters. To make them powerful enough to represent complicated distributions (i.e., go from the limited parametric setting to a non-parametric one), we consider that some of the variables are never observed (they are called hidden). By having more hidden variables (also called hidden units), we can increase the modeling capacity of the Boltzmann Machine (BM). Restricted Boltzmann Machines further restrict BMs to those without visible-visible and hidden-hidden connections. A graphical depiction of an RBM is shown below.

2008

• (Larochelle & Bengio, 2008) ⇒ H. Larochelle, and Yoshua Bengio. (2008). “Classification using discriminative restricted Boltzmann machines". In: Proceedings of the 25th International Conference on Machine learning (ICML 2008). doi:10.1145/1390156.1390224

2006

• (Hinton & Salakhutdinov, 2006) ⇒ Geoffrey E. Hinton, and Ruslan R. Salakhutdinov. (2006). “Reducing the Dimensionality of Data with Neural Networks.” In: Science, 313(5786). doi:10.1126/science.1127647
  • QUOTE: An ensemble of binary vectors (e.g., images) can be modeled using a two-layer network called a "restricted Boltzmann machine" (RBM) (5, 6) in which stochastic, binary pixels are connected to stochastic, binary feature detectors using symmetrically weighted connections. The pixels correspond to "visible" units of the RBM because their states are observed; the feature detectors correspond to "hidden" units. A joint configuration [math]\displaystyle{ (\mathbf{v}, \mathbf{h}) }[/math] of the visible and hidden units has an energy (7) given by...

---