Unsmoothed Maximum Likelihood-based Character-Level Language Modeling Algorithm

An Unsmoothed Maximum Likelihood-based Character-Level Language Modeling Algorithm is a MLE-based character-level language modeling algorithm that is a unsmoothed MLE algorithm.

• Example(s):
  • the one presented in (Goldberg, 2015).
  • …
• Counter-Example(s):
  • an LSTM-based Character-Level Language Modeling Algorithm.
  • a Smoothed Maximum Likelihood Character Level Language Model.
• See: MLE-based Language Model.

---

---

References

2015

• http://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139
  • QUOTE: ... I will briefly describe these character-level maximum-likelihood language models, which are much less magical than RNNs and LSTMs, ...

    ... Mathematically, we would like to learn a function P(c|h). Here, c is a character, h is a n-letters history, and P(c|h) stands for how likely is it to see c after we've seen h.

    Perhaps the simplest approach would be to just count and divide (a.k.a maximum likelihood estimates). We will count the number of times each letter c′ appeared after h, and divide by the total numbers of letters appearing after h. The unsmoothed part means that if we did not see a given letter following h, we will just give it a probability of zero.

---