Neural-based Character-Level Language Model (LM)

A Neural-based Character-Level Language Model (LM) is a character-level LM that is a neural LM.

• Context:
  • It can (typically) be produced by a Neural Character-Level Language Model Training System.
• Example(s):
  • an LSTM-based Character-Level Language Model, as by (Karpathy, 2015).
  • …
• Counter-Example(s):
  • A Neural-based Word/Token-Level Language Model (LM).
  • A MLE-based Character-Level Language Model (LM).
• See: Maximum Likelihood-based Character-Level Language Model.

---

---

References

2015

• (Karpathy, 2015) ⇒ Andrej Karpathy. (2015). “The Unreasonable Effectiveness of Recurrent Neural Networks.” In: Proceedings of Blog post 2015-05-21.
  • QUOTE: ... By the way, together with this post I am also releasing code on Github that allows you to train character-level language models based on multi-layer LSTMs. You give it a large chunk of text and it will learn to generate text like it one character at a time. ...

    

    An example RNN with 4-dimensional input and output layers, and a hidden layer of 3 units (neurons). This diagram shows the activations in the forward pass when the RNN is fed the characters "hell" as input. The output layer contains confidences the RNN assigns for the next character (vocabulary is "h,e,l,o"); We want the green numbers to be high and red numbers to be low.

---