Attention Module

An Attention Module is a neural network module that uses an alignment score function to amplify some parts of the input data while diminishing other parts.

• Context:
  • It can range from being a Self-Attention Module, Hard Attention Module to being a Soft Attention Module.
  • …
• Example(s):
  • Scaled Dot Product-based Attention.
  • Query-Key-Value-based Attention, composed of a Query Matrix (Q), Key Matrix (K) and a Value Matrix V
  • hard, soft, self, cross, Luong, and Bahdanau
  • …
• Counter-Example(s):
  • a Coverage Mechanism Module,
  • a Gating Mechanism Module.
• See: Attention Mechanism, Transformer-based NNet, Differentiable Neural Computer.

---

---

References

2022

• (Wikipedia, 2022) ⇒ https://en.wikipedia.org/wiki/Attention_(machine_learning) Retrieved:2022-4-24.
  • In neural networks, attention is a technique that mimics cognitive attention. The effect enhances some parts of the input data while diminishing other parts — the thought being that the network should devote more focus to that small but important part of the data. Learning which part of the data is more important than others depends on the context and is trained by gradient descent.

    Attention-like mechanisms were introduced in the 1990s under names like multiplicative modules, sigma pi units, and hypernetworks. Its flexibility comes from its role as "soft weights" that can change during runtime, in contrast to standard weights that must remain fixed at runtime. Uses of attention include memory in neural turing machines, reasoning tasks in differentiable neural computers, language processing in transformers, and multi-sensory data processing (sound, images, video, and text) in perceivers.

2022

• (Wikipedia, 2022) ⇒ https://en.wikipedia.org/wiki/Attention_(machine_learning)#Variants Retrieved:2022-4-24.
  • There are many variants of attention: dot product, query-key-value, hard, soft, self, cross, Luong, and Bahdanau to name a few. These variants recombine the encoder-side inputs to redistribute those effects to each target output. Often, a correlation-style matrix of dot products provides the re-weighting coefficients (see legend).

2018

• https://lilianweng.github.io/posts/2018-06-24-attention/
  • QUOTE: Below is a summary table of several popular attention mechanisms and corresponding alignment score functions:

Content-base attention

Below is a summary table of several popular attention mechanisms and corresponding alignment score functions:

Name	Alignment score function	Citation

Name	Alignment score function	Citation
Content-base attention	$\text{score}({\mathit{s}}_t,{\mathit{h}}_i)=\text{cosine}[{\mathit{s}}_t,{\mathit{h}}_i]$	Graves2014
Additive(*)	$\text{score}({\mathit{s}}_t,{\mathit{h}}_i)={\mathbf{v}}_a^{\mathrm{\top }}\tanh ({\mathbf{W}}_a[{\mathit{s}}_t;{\mathit{h}}_i])$	Bahdanau2015
Location-Base	${\alpha }_{t,i}=\text{softmax}({\mathbf{W}}_a{\mathit{s}}_t)$ Note: This simplifies the softmax alignment to only depend on the target position.	Luong2015
General	$\text{score}({\mathit{s}}_t,{\mathit{h}}_i)={\mathit{s}}_t^{\mathrm{\top }}{\mathbf{W}}_a{\mathit{h}}_i$ where ${\mathbf{W}}_a$ is a trainable weight matrix in the attention layer.	Luong2015
Dot-Product	$\text{score}({\mathit{s}}_t,{\mathit{h}}_i)={\mathit{s}}_t^{\mathrm{\top }}{\mathit{h}}_i$	Luong2015
Scaled Dot-Product(^)	$\text{score}({\mathit{s}}_t,{\mathit{h}}_i)=\frac{{\mathit{s}}_t^{\mathrm{\top }}{\mathit{h}}_i}{\sqrt{n}}$ Note: very similar to the dot-product attention except for a scaling factor; where n is the dimension of the source hidden state.	Vaswani2017

(*) Referred to as “concat” in Luong, et al., 2015 and as “additive attention” in Vaswani, et al., 2017.
(^) It adds a scaling factor $1/\sqrt{n}$, motivated by the concern when the input is large, the softmax function may have an extremely small gradient, hard for efficient learning.

Here are a summary of broader categories of attention mechanisms:

Name	Definition	Citation
Self-Attention(&)	Relating different positions of the same input sequence. Theoretically the self-attention can adopt any score functions above, but just replace the target sequence with the same input sequence.	Cheng2016
Global/Soft	Attending to the entire input state space.	Xu2015
Local/Hard	Attending to the part of input state space; i.e. a patch of the input image.	Xu2015; Luong2015

---