Hierarchical Attention Network (HAN)

A Hierarchical Attention Network (HAN) is an Attentional Neural Network that includes a hierarchical attention mechanism.

• Example(s):
  • a Hierarchical Attention Network for Text Classification (Yang et al., 2016).
  • …
• Counter-Example(s):
  • a Hierarchical Bayesian Network,
  • a Hierarchical Recurrent Encoder-Decoder (HRED) Neural Network,
  • a Pointer Network (Ptr-Net),
  • a Pointer-Generator Network,
  • a Sequence-to-Sequence Neural Network with Attention Mechanism,
  • a Transformer Neural Network.
• See: Long Short-Term Memory, Recurrent Neural Network, Convolutional Neural Network, Gating Mechanism.

---

---

References

2016

• (Yang et al., 2016) ⇒ Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. (2016). “Hierarchical Attention Networks for Document Classification.” In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-2016).
  • QUOTE: We propose a hierarchical attention network for document classification. Our model has two distinctive characteristics: (i) it has a hierarchical structure that mirrors the hierarchical structure of documents; (ii) it has two levels of attention mechanisms applied at the word and sentence-level, enabling it to attend differentially to more and less important content when constructing the document representation(...)

    The overall architecture of the Hierarchical Attention Network (HAN) is shown in Fig. 2. It consists of several parts: a word sequence encoder, a word-level attention layer, a sentence encoder and a sentence-level attention layer. We describe the details of different components in the following sections.

Figure 2: Hierarchical Attention Network.

---