Attention-based Encoder-Decoder Network

A Attention-based Encoder-Decoder Network is a neural encoder-decoder network that includes an attention mechanism.

• Context:
  • It can (typically) be used in tasks involving sequence-to-sequence (seq2seq) modeling, such as machine translation, speech recognition, and text summarization.
  • It can (often) improve the performance of traditional seq2seq models by focusing on relevant parts of the input sequence for each step of the output sequence.
  • It can include various forms of attention mechanisms, such as global attention, local attention, and multi-head attention.
  • ...
• Example(s):
  • the Transformer Neural Network, which uses a self-attention mechanism (Vaswani et al., 2017),
  • the Neural Machine Translation by Jointly Learning to Align and Translate model (Bahdanau et al., 2015).
  • the Pointer-Generator Seq2Seq Neural Network (See et al., 2017).
  • ...
• Counter-Example(s):
  • a Bidirectional Associative Memory (BAM) Network,
  • a LSTM-based Encoder/Decoder Neural Network,
  • a Sequence-to-Sequence (Seq2Seq) Neural Network without Attention.
• See: Sequence-to-Sequence Model, Neural Machine Translation, Recurrent Encoder-Decoder Neural Network, Seq2Seq with Attention Training Algorithm, Artificial Neural Network, Natural Language Processing Task, Language Model.

---

---

---

---