LSTM-based Language Modeling System

A LSTM-based Language Modeling System is a neural LM system that implements an LSTM-based LM algorithm to solve an LSTM-based language modeling task (that requires an LSTM-based language models).

• Context:
  • It can be a PyTorch LSTM-based Language Language Modeling System.
• Example(s):
  • …
• Counter-Example(s):
  • an seq2seq-based Modeling System.
  • a Transformer Network-based LM System.
• See: LSTM Algorithm, PyTorch, Python-based Language Modeling System.

---

---

References

2019

• Rani Horev. (2019). “Transformer-XL Explained: Combining Transformers and RNNs into a State-of-the-art Language Model."
  • QUOTE: ... A popular approach for language modeling is Recurrent Neural Networks (RNNs) as they capture dependencies between words well, especially when using modules such as LSTM. However, RNNs tend to be slow and their ability to learn long-term dependencies is still limited due to vanishing gradients. Transformers, invented in 2017, introduced a new approach — attention modules. Instead of processing tokens one by one, attention modules receive a segment of tokens and learn the dependencies between all of them at once …

---