Recurrent Neural Network Unit

A Recurrent Neural Network Unit is a Neural Network Unit that is composed by a feedback loop and is part of a recurrent neural network.

• AKA: Recurrent Unit.
• Example(s):
  • a GRU,
  • a LSTM Unit.
  • …
• Counter-Example(s):
  • a Max-Pooling Unit,
  • a Sigmoid Unit.
• See: Artificial Neuron, Neural Network Layer, Memory Cell, Convolutional Neural Network, Feedforward Neural Network.

---

---

References

2014

• (Chung et al., 2014) ⇒ Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. (2014). “Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling.” In: eprint arXiv:1412.3555.
  • QUOTE: The Long Short-Term Memory (LSTM) unit was initially proposed by Hochreiter and Schmidhuber (1997). Since then, a number of minor modifications to the original LSTM unit have been made. We follow the implementation of LSTM as used in Graves (2013) (...)

    See Fig.1 (a) for the graphical illustration.

Figure 1: Illustration of (a) LSTM and (b) gated recurrent units. (a) $i$, $f$ and $o$ are the input, forget and output gates, respectively. $c$ and $\tilde{c}$ denote the memory cell and the new memory cell content. (b) $r$ and $z$ are the reset and update gates, and $h$ and $\tilde{h}$ are the activation and the candidate activation.

2013a

• (Grossberg,2013) ⇒ Stephen Grossberg (2013), Recurrent neural networks"Scholarpedia, 8(2):1888. doi:10.4249/scholarpedia.1888
  • QUOTE: A recurrent neural network (RNN) is any network whose neurons send feedback signals to each other. This concept includes a huge number of possibilities. A number of reviews already exist of some types of RNNs. These include [1], [2], [3], [4].

    Typically, these reviews consider RNNs that are artificial neural networks (aRNN) useful in technological applications. To complement these contributions, the present summary focuses on biological recurrent neural networks (bRNN) that are found in the brain. Since feedback is ubiquitous in the brain, this task, in full generality, could include most of the brain's dynamics. The current review divides bRNNS into those in which feedback signals occur in neurons within a single processing layer,  which occurs in networks for such diverse functional roles as storing spatial patterns in short-term memory, winner-take-all decision making, contrast enhancement and normalization, hill climbing, oscillations of multiple types (synchronous, traveling waves, chaotic), storing temporal sequences of events in working memory, and serial learning of lists; and those in which feedback signals occur between multiple processing layers, such as occurs when bottom-up adaptive filters activate learned recognition categories and top-down learned expectations focus attention on expected patterns of critical features and thereby modulate both types of learning.

2013b

• (Graves et al., 2013) ⇒ Alex Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. (2013). “Hybrid Speech Recognition with Deep Bidirectional LSTM.” In: Proceedings of 2013 IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU-2013).
  • QUOTE: Given an input sequence [math]\displaystyle{ x = (x_1, ..., x_T) }[/math], a standard recurrent neural network (RNN) computes the hidden vector sequence [math]\displaystyle{ h = (h_1, ..., h_T) }[/math] and output vector sequence [math]\displaystyle{ y = (y_1, ..., y_T) }[/math] by iterating the following equations from t = 1 to T:
    [math]\displaystyle{ h_t = \mathcal{H}(W_xh_{xt} + W_{hh}h_{t-1} + b_h) (1) }[/math]
    [math]\displaystyle{ y_t = W_hy_{ht} + b_y (2) }[/math]
    where the W terms denote weight matrices (e.g. Wxh is the input-hidden weight matrix), the b terms denote bias vectors (e.g. bh is hidden bias vector) and H is the hidden layer function.

2011

• (Sammut & Webb, 2011) ⇒ Claude Sammut, and Geoffrey I. Webb. (2011). “Simple Recurrent Network.” In: (Sammut & Webb, 2011) p.906
  • QUOTE: The simple recurrent network is a specific version of the backpropagation neural network that makes it possible to process sequential input and output (Elman 1990). It is typically a three-layer network where a copy of the hidden layer activations is saved and used (in addition to the actual input) as input to the hidden layer in the next time step. The previous hidden layer is fully connected to the hidden layer. Because the network has no recurrent connections per se (only a copy of the activation values), the entire network (including the weights from the previous hidden layer to the hidden layer) can be trained with the backpropagation algorithm as usual. It can be trained to read a sequence of inputs into a target output pattern, to generate a sequence of outputs from a given input pattern, or to map an input sequence to an output sequence (as in predicting the next input). Simple recurrent networks have been particularly useful in time series prediction, as well as in modeling cognitive processes, such as language understanding and production.

2005

• (Golda, 2005) ⇒ Adam Golda (2005). "Introduction to neural networks"
  • QUOTE: Recurrent neural networks. Such networks have feedback loops (at least one) output signals of a layer are connected to its inputs. It causes dynamic effects during network work. Input signals of layer consist of input and output states (from the previous step) of that layer. The structure of recurrent network depicts the below figure.

    

---