Hierarchical Recurrent Encoder-Decoder (HRED) Neural Network

A Hierarchical Recurrent Encoder-Decoder (HRED) Neural Network is an Encoder-Decoder Neural Network that can generate query suggestions by computing session-level and query-level recurrent states.

• Example(s):
  • Hierarchical Recurrent Encoder-Decoder Neural Network for Query Suggestion (Sordoni et al., 2015),
  • …
• Counter-Example(s):
  • a Hierarchical Attention Network,
  • a Hierarchical Bayesian Network,
  • a Pointer Network (Ptr-Net),
  • a Pointer-Generator Network,
  • a Sequence-to-Sequence Neural Network with Attention Mechanism,
  • a Transformer Neural Network.
• See: Long Short-Term Memory, Recurrent Neural Network, Convolutional Neural Network, Gating Mechanism.

---

---

References

2015

• (Sordoni et al., 2015) ⇒ Alessandro Sordoni, Yoshua Bengio, Hossein Vahabi, Christina Lioma, Jakob Grue Simonsen, and Jian-Yun Nie. (2015). “A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion.” In: Proceedings of the 24th ACM International Conference on Information and Knowledge Management (CIKM 2015). DOI:10.1145/2806416.2806493. arXiv:1507.02221.
  • QUOTE: Our hierarchical recurrent encoder-decoder (HRED) is pictured in Figure 3. Given a query in the session, the model encodes the information seen up to that position and tries to predict the following query. The process is iterated throughout all the queries in the session. In the forward pass, the model computes the query-level encodings, the session-level recurrent states and the log-likelihood of each query in the session given the previous ones. In the backward pass, the gradients are computed and the parameters are updated.

Figure 3: The hierarchical recurrent encoder-decoder (HRED) for query suggestion. Each arrow is a non-linear transformation. The user types cleveland gallery → lake erie art. During training, the model encodes cleveland gallery, updates the session-level recurrent state and maximize the probability of seeing the following query lake erie art. The process is repeated for all queries in the session. During testing, a contextual suggestion is generated by encoding the previous queries, by updating the session-level recurrent states accordingly and by sampling a new query from the last obtained session-level recurrent state. In the example, the generated contextual suggestion is cleveland indian art.

---