Recurrent Neural Network (RNN)-based Language Model Training Algorithm

A Recurrent Neural Network (RNN)-based Language Model Training Algorithm is a neural LM algorithm that includes an RNN training algorithm.

• Context:
  • It can be implemented by an RNN-based LM Training System (to solve an RNN-based LM training task).
• Example(s):
  • an LSTM-based LM Algorithm.
  • a Transformer-based LM Algorithm.
  • …
• Counter-Example(s):
  • a Transformer-based LM Training Algorithm (for a transformer-based LM model).
  • an MLE-based LM Algorithm.
• See: Text-String Likelihood Scoring Function Training Algorithm.

---

---

References

2016

• (Kim et al., 2016) ⇒ Yoon Kim, Yacine Jernite, David Sontag, and Alexander M. Rush. (2016). “Character-Aware Neural Language Models.” In: Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (AAAI-2016).
  • QUOTE: ... We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language model (RNN-LM). ...

---