Text Tokenization Algorithm

A Text Tokenization System is a text segmentation algorithm/string segmentation algorithm that can be applied by a text tokenization system (to solve a text tokenization task).

• See: End-of-Sentence Detection Algorithm, Word Segmentation Algorithm.

---

---