Encoder-Decoder Neural Network

An Encoder-Decoder Neural Network is a modular neural network that consists of an encoder network and a decoder network.

Context:
- It can range from being a Encoder-Decoder CNN, to being a Encoder-Decoder RNN, to being a Encoder-Decoder Network RNN-CNN.
- It can be trained by a Neural Encoder-Decoder Training System (that implements a neural encoder-decoder training algorithm).
- It can be used in Sequence-to-Sequence Learning Tasks.
- …
Example(s):
- a Neural Auto-Encoding Network;
- a Recurrent (RNN/RNN)-based Encoder-Decoder Neural Network such as: an Attention-based Encoder-Decoder RNN and a DeepFix Seq2Seq Neural Network.
- an Attention-based Encoder-Decoder Network.
- a Convolutional (CNN/CNN)-based Encoder-Decoder Neural Network.
- a GAP Encoder-Decoder Neural Network.
- a Seq2Seq Encoder-Decoder Neural Network.
- a Transliteration Encoder-Decoder Neural Network.
- a Spelling Correction Encoder-Decoder Neural Network.
- a WikiText Error Correction Encoder-Decoder Neural Network such as: GM-RKB Seq2Seq WikiText Error Correction Neural Network.
- Text-to-Text Encoder-Decoder Model: T5 and BART for machine translation and summarization tasks.
- Speech-to-Text Encoder-Decoder Model (speech-to-text neural network model).
- …
Counter-Example(s):
See: Autoencoder Network, Sequence Learning, Memory Augmented Neural Network Training System, Transformer-based NNet.

References

2018a

(Liao et al., 2018) ⇒ Binbing Liao, Jingqing Zhang, Chao Wu, Douglas McIlwraith, Tong Chen, Shengwen Yang, Yike Guo, and Fei Wu. (2018). “Deep Sequence Learning with Auxiliary Information for Traffic Prediction.” In: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ISBN:978-1-4503-5552-0 doi:10.1145/3219819.3219895
- QUOTE: In this paper, we effectively utilise three kinds of auxiliary information in an encoder-decoder sequence to sequence (Seq2Seq) [7, 32] learning manner as follows: a wide linear model is used to encode the interactions among geographical and social attributes, a graph convolution neural network is used to learn the spatial correlation of road segments, and the query impact is quantified and encoded to learn the potential influence of online crowd queries(...)
  Figure 4 shows the architecture of the Seq2Seq model for traffic prediction. The encoder embeds the input traffic speed sequence [math]\displaystyle{ \{v_1,v_2, \cdots, v_t \} }[/math] and the final hidden state of the encoder is fed into the decoder, which learns to predict the future traffic speed [math]\displaystyle{ \{\tilde{v}_{t+1},\tilde{v}_{t+2}, \cdots,\tilde{v}_{t+t'} \} }[/math]. Hybrid model that integrates the auxiliary information will be proposed based on the Seq2Seq model.
  
  Figure 4: Seq2Seq: The Sequence to Sequence model predicts future traffic speed [math]\displaystyle{ \{\tilde{v}_{t+1},\tilde{v}_{t+2}, \cdots ,\tilde{v}_{t+t'} \} }[/math], given the previous traffic speed [math]\displaystyle{ {v_1,v_2, ...v_t } }[/math].

2018b

(Saxena, 2018) ⇒ Rohan Saxena (April, 2018). "What is an Encoder/Decoder in Deep Learning?". In: Quora.
- QUOTE: Some network architectures explicitly aim to leverage this ability of neural networks to learn efficient representations. They use an encoder network to map raw inputs to feature representations, and a decoder network to take this feature representation as input, process it to make its decision, and produce an output. This is called an encoder-decoder network.
  ...
  In a CNN, an encoder-decoder network typically looks like this (a CNN encoder and a CNN decoder):
  Image Credits: (Badrinarayanan et al., 2017)
  ...
  In an RNN, an encoder-decoder network typically looks like this (an RNN encoder and an RNN decoder):
  ...
  We can mix and match and use a CNN encoder with RNN decoder, or vice versa; any combination can be used which is suitable for the given task.
  An example from a paper I recently implemented from scratch, Text-guided Attention Model for Image Captioning (T-ATT):

2017a

(Ramachandran et al., 2017) ⇒ Prajit Ramachandran, Peter J. Liu, and Quoc V. Le. (2017). “Unsupervised Pretraining for Sequence to Sequence Learning.” In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). arViv:1611.02683
- QUOTE: Therefore, the basic procedure of our approach is to pretrain both the seq2seq encoder and decoder networks with language models, which can be trained on large amounts of unlabeled text data. This can be seen in Figure 1, where the parameters in the shaded boxes are pretrained. In the following we will describe the method in detail using machine translation as an example application.
  
  Figure 1: Pretrained sequence to sequence model. The red parameters are the encoder and the blue parameters are the decoder. All parameters in a shaded box are pretrained, either from the source side (light red) or target side (light blue) language model. Otherwise, they are randomly initialized.

2017b

(Badrinarayanan et al., 2017) ⇒ Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. (2017). “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation.” In: IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12). doi:10.1109/TPAMI.2016.2644615

2016a

(Cucurull, 2016) ⇒ Guillem Cucurull (May 3, 2016). “What is the SegNet neural network? Why is it important?."
- QUOTE: It is a convolutional neural network (CNN) that performs image segmentation. This means that the network learns to assign each pixel a class depending on the object or surface it belongs, e.g a car, highway, tree, building...
  It uses an Encoder-Decoder architecture, were the image is first down-sampled by an encoder as in a "traditional" CNN like VGG, and then it is upsampled by using a decoder that is like a reversed CNN, with upsampling layers instead of pooling layers.

2016b

(Schnober et al., 2016) ⇒ Carsten Schnober, Steffen Eger, Erik-Lan Do Dinh, and Iryna Gurevych. (2016). “Still Not There? Comparing Traditional Sequence-to-Sequence Models to Encoder-Decoder Neural Networks on Monotone String Translation Tasks.” In: Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.
- QUOTE: We compare three variants of encoder-decoder models: the ‘classic’ variant and two modifications:
  - enc-dec: Encoder-decoder models using recurrent neural networks (RNNs) for Seq2Seq tasks were introduced by Cho et al. (2014) and Sutskever et al. (2014) (...)
  - attn-enc-dec: We explore the attention-based encoder-decoder model proposed by Bahdanau et al. (2014) (Figure 1). It extends the encoder-decoder model by learning to align and translate jointly (...)
  - morph-trans: Faruqui et al. (2016) present a new encoder-decoder model designed for morphological inflection, proposing to feed the input sequence directly into the decoder. This approach is motivated by the observation that input and output are usually very similar in problems such as morphological inflection (...)

Figure 1: In the encoder-decoder model, the encoder (bottom) generates a representation of the input sequence $\vec{x}$ from which the decoder (top) generates the output sequence $\vec{y}$. The attention-based mechanism (shown here) enables the decoder to “peek" into the input at every decoding step through multiple input representations $a_t$. Illustration from Bahdanau et al. (2014).

2016c

(Faruqui et al., 2016) ⇒ Manaal Faruqui, Yulia Tsvetkov, Graham Neubig, and Chris Dyer. (2016). “Morphological Inflection Generation Using Character Sequence to Sequence Learning.” In: Proceedings of the 2016 Conference of the North {{American Chapter of the Association for Computational Linguistics: Human Language Technologies. DOI:10.18653/v1/N16-1077

Encoder-Decoder Neural Network

References

2018a

2018b

2017a

2017b

2016a

2016b

2016c

2016d

2015

2014a

2014b

Navigation menu

Search