Hopfield Recurrent Neural Network

A Hopfield Recurrent Neural Network is a recurrent neural network with based on associative memories and Lyapunov functions.

• Context:
  • It can (typically) store and retrieve memory patterns through an Associative Memory Process.
  • It can (typically) be produced by Hopfield Network Training System (that solves a Hopfield Network Training System Task by implementing a Hopfield Network Training Algorithm).
  • It can (typically) use symmetric weight matrices to ensure convergence to stable states.
  • It can (typically) perform as an Autoassociative Memory Model, where the network can recall entire patterns from partial or noisy input.
  • It can (often) converge to a stable state that corresponds to a local minimum of the energy function (defined by a Lyapunov Function).
  • It can range from being a Binary Hopfield Network to being a Continuous Hopfield Network.
  • …
• Example(s):
  • as proposed in (Hopfield (1982).
  • Ramsaeur-Hopflied Neural Network :hopfield-layers (Ramsauer et al., 2020),
  • hopfieldnetwork 1.0.1 - hopfield network's python implementation by Felix Andreas (2020),
  • …
• Counter-Example(s):
  • a Feedforward network,
  • a Radial-Basis Function Network,
  • a Bidirectional Associative Memory (BAM) Network,
  • a DAG Recurrent Neural Network,
  • an Echo State Network,
  • an Elman Network,
  • a Gated Recurrent Unit (GRU) Network,
  • a Jordan Network,
  • a Long Short-Term Memory (LSTM) Network,
  • a Recurrent Multilayer Perceptron Network,
  • a Recursive Neural Network,
  • a Restricted Boltzmann Machine.
• See: Content-Addressable Memory, Associative Memory, Attractor Network, Boltzmann Machine, Dynamical Systems, Oscillatory Associative Memory, Simulated Annealing.

---

---

References

2020

• (Ramsauer et al., 2020) ⇒ Hubert Ramsauer, Bernhard Schafl, Johannes Lehner, Philipp Seidl, Michael Widrich, Thomas Adler, Lukas Gruber, Markus Holzleitner, Milena Pavlovic, Geir Kjetil Sandve, Victor Greiff, David Kreil, Michael Kopp, Gunter Klambauer, Johannes Brandstetter, and Sepp Hochreiter (2020)."Hopfield Networks is All You Need". In: arXiv preprint arXiv:2008.02217.
  • QUOTE: We introduce a modern Hopfield network with continuous states and a corresponding update rule. The new Hopfield network can store exponentially (with the dimension of the associative space) many patterns, retrieves the pattern with one update, and has exponentially small retrieval errors.

2017

• (Miikkulainen,2017) ⇒ Miikkulainen R. (2017) "Hopfield Network". In: Sammut, C., Webb, G.I. (eds) Encyclopedia of Machine Learning and Data Mining. Springer, Boston, MA
  • QUOTE: The Hopfield network is a binary, fully recurrent network that, when started on a random activation state, settles the activation over time into a state that represents a solution (Hopfield and Tank 1986). This architecture has been analyzed thoroughly using tools from statistical physics. In particular, with symmetric weights, no self-connections, and asynchronous neuron activation updates, a Lyapunov function exists for the network, which means that the network activity will eventually settle. The Hopfield network can be used as an associate memory or as a general optimizer. When used as an associative memory, the weight values are computed from the set of patterns to be stored. During retrieval, part of the pattern to be retrieved is activated, and the network settles into the complete pattern. When used as an optimizer, the function to be optimized is mapped into the Lyapunov function of the network, which is then solved for the weight values. The network then settles to a state that represents the solution. The basic Hopfield architecture can be extended in many ways, including continuous neuron activations. However, it has limited practical value mostly because it is not strong in either of the above task: as an associative memory, its capacity is approximately 0.15N in practice (where N is the number of neurons), and as an optimizer, it often settles into local optima instead of the global one. The Boltzmann machine extends the architecture with hidden neurons, allowing for better performance in both tasks. However, the Hopfield network has had a large impact in the field because the theoretical techniques developed for it have inspired theoretical approaches for other architectures as well, especially for those of self-organizing systems (e.g., self-organizing maps, adaptive resonance theory).

2013

• (Kaushik, 2013) ⇒ Saroj Kaushik. (2013). “Artificial Neural Network - Lecture Model 22”. Course Material
  • QUOTE: A Hopfield network is a kind of recurrent network as output values are fed back to input in an undirected way.
    • It consists of a set of N connected neurons with weights which are symmetric and no unit is connected to itself.
    • There are no special input and output neurons.
    • The activation of a neuron is binary value decided by the sign of the weighted sum of the connections to it.
    • A threshold value for each neuron determines if it is a firing neuron.
    • A firing neuron is one that activates all neurons that are connected to it with a positive weight.
    • The input is simultaneously applied to all neurons, which then output to each other.
    • This process continues until a stable state is reached.

2012

• (Wikiepdia, 2021) ⇒ http://en.wikipedia.org/wiki/Hopfield_network
  • QUOTE: A Hopfield network is a form of recurrent artificial neural network invented by John Hopfield. Hopfield nets serve as content-addressable memory systems with binary threshold units. They are guaranteed to converge to a local minimum, but convergence to one of the stored patterns is not guaranteed. Furthermore, Hopfield networks provide a model for understanding human memory.

2007

• (Hopfield, 2007) ⇒ John J. Hopfield (2007), "Hopfieldnetwork" In: Scholarpedia, 2(5):1977. doi:10.4249/scholarpedia.1977
  • QUOTE: A Hopfield net is a recurrent neural network having synaptic connection pattern such that there is an underlying Lyapunov function for the activity dynamics. Started in any initial state, the state of the system evolves to a final state that is a (local) minimum of the Lyapunov function.

There are two popular forms of the model:

• Binary neurons with discrete time, updated one at a time

  [math]\displaystyle{ V_j(t+1) = \begin{cases} 1, & \mbox{ if } \ \Sigma_k T_{jk}V_k(t) + I_j\gt 0 \\ 0, & \mbox{ otherwise } \end{cases} }[/math]

• Graded neurons with continuous time

  [math]\displaystyle{ dx_j/dt = -x_j/\tau + \Sigma_k T_{jk}g(x_k) + I_j\ . }[/math]

Here,

• [math]\displaystyle{ V_j }[/math] denotes activity of the [math]\displaystyle{ j }[/math]-th neuron.
• [math]\displaystyle{ x_j }[/math] is the mean internal potential of the neuron.
• [math]\displaystyle{ I_j }[/math] is direct input (e.g., sensory input or bias current) to the neuron.
• [math]\displaystyle{ T_{jk} }[/math] is the strength of synaptic input from neuron [math]\displaystyle{ k }[/math] to neuron [math]\displaystyle{ j\ . }[/math]
• [math]\displaystyle{ g }[/math] is a monotone function that converts internal potential into firing rate output of the neuron, i.e., [math]\displaystyle{ V_j=g(x_j)\ . }[/math].

1982

• (Hopfield, 1982) ⇒ J. J. Hopfield. (1982). “Neural Networks and Physical systems with emergent collective computational abilities.” In: Proceedings of the National Academy of Sciences of the USA, vol. 79 no. 8.

---