Neural Hidden State

A Neural Hidden State is a output function of a hidden neuron.

• AKA: Hidden State, Hidden Neuron State Function.
• Context:
  • It can be defined as [math]\displaystyle{ h =g(W, x, \Theta) }[/math], where [math]\displaystyle{ g }[/math] is a neuron activation function, [math]\displaystyle{ W }[/math] is a weight matrix, [math]\displaystyle{ x_i }[/math] are neural inputs, [math]\displaystyle{ \Theta }[/math] represents other state functions or variables.
• Example(s):
  • a simple Feedforward Hidden State: [math]\displaystyle{ h=g(Wx+b) }[/math],
  • a Recurrent Hidden State: [math]\displaystyle{ h_t=g(Wx_{t-1}+U h_{t-1}) }[/math] ,
  • …
• Counter-Example(s)
  • a Neuron Activation Function.
  • a Neural Input,
  • a Neural Ouput,
  • a Neural Network Bias.
  • a Neural Network Weight.
• See: Visible Neural Network Unit, Artificial Neural Network, Artificial Neuron, Fully-Connected Neural Network Layer, Neuron Activation Function, Neural Network Weight, Neural Network Connection, Neural Network Topology, Multi Hidden Layer NNet.

---

---

References

2018

• (Jurasky & Martin, 2018) ⇒ Daniel Jurafsky, and James H. Martin (2018). "Chapter 9 -- Sequence Processing with Recurrent Networks". In: Speech and Language Processing (3rd ed. draft). Draft of September 23, 2018.
  • QUOTE: Fortunately, this complexity is largely encapsulated within the basic processing units, allowing us to maintain modularity and to easily experiment with different architectures. To see this, consider Fig. 9.14 which illustrates the inputs/outputs and weights associated with each kind of unit.
    

    Figure 9.14 Basic neural units used in feed-forward, simple recurrent networks (SRN), long short-term memory (LSTM) and gate recurrent units

    The neural units used in LSTMs and GRUs are obviously much more complex than basic feed-forward networks.

    At the far left, (a) is the basic feed-forward unit [math]\displaystyle{ h = g(W x+b) }[/math]. A single set of weights and a single activation function determine its output, and when arranged in a layer there is no connection between the units in the layer. Next, (b) represents the unit in an SRN. Now there are two inputs and additional set of weights to go with it. However, there is still a single activation function and output. When arranged as a layer, the hidden layer from each unit feeds in as an input to the next (...)

2017

• (Dey & Salem, 2017) ⇒ Rahul Dey, and Fathi M. Salem (2017). "Gate-variants of Gated Recurrent Unit (GRU) neural networks" In: 2017 IEEE 60th International Midwest Symposium on Circuits and Systems (MWSCAS), Boston, MA, 2017, pp. 1597-1600. doi: 10.1109/MWSCAS.2017.8053243
  • QUOTE: : In principal, RNN are more suitable for capturing relationships among sequential data types. The so-called simple RNN has a recurrent hidden state as in
    [math]\displaystyle{ h_t=g(Wx_t+Uh_{t-1})+b\quad }[/math](1)

    where [math]\displaystyle{ x_t }[/math] is the (external) m-dimensional input vector at time [math]\displaystyle{ t }[/math], [math]\displaystyle{ h_t }[/math] the n-dimensional hidden state, [math]\displaystyle{ g }[/math] is the (point-wise) activation function, such as the logistic function, the hyperbolic tangent function, or the rectified Linear Unit (ReLU) [2, 6], and [math]\displaystyle{ W }[/math], [math]\displaystyle{ U }[/math] and [math]\displaystyle{ b }[/math] are the appropriately sized parameters (two weights and bias). Specifically, in this case, [math]\displaystyle{ W }[/math] is an [math]\displaystyle{ n\times m }[/math] matrix, [math]\displaystyle{ U }[/math] is an [math]\displaystyle{ n\times n }[/math] matrix, and [math]\displaystyle{ b }[/math] is an [math]\displaystyle{ n\times 1 }[/math] matrix (or vector).

---