Text Segmentation System

A Text Segmentation System is a text processing system that applies a text segmentation algorithm to solve a text segmentation task.

• AKA: Automatic Text Chunking System.
• Context:
  • It can range from being a Shallow Parsing System, Phrase Chunking System, Word Mention Recognition System, to being a Heuristic Text Segmentation System or a Statistical Text Segmentation System.
  • It can be a Text with Overlaps Segmentation System (for text segmentation with overlaps).
  • ...
• Example(s):
  • Sentence Segmentation Systems that performs sentence segmentation.
  • Text Tokenization System, such as: a Word Mention Segmentation System.
  • Phrase Chunking System, such as: an NP chunking system (such as YamCha for noun phrase chunking).
  • Paragraph Chunking System, such as a ...
  • Platform-based Text Segmenter, such as:
    • an NLTK-based Text Segmenter,
    • a LingPipe-based Text Segmenter,
    • a LangChain-based Text Splitter.
    • ...
• Counter-Example(s):
  • a Text Item Classification System, such as a POS Tagging System, or a Named Entity Recognition System.
  • a Text Item Clustering System.
  • a DNA Chunking System,
  • a Base Chunking System,
• See: NLP System, Morphological Analysis System, Segmentation System, Text Mining System, Sequence-to-Sequence Neural Network, Natural Language Processing System, Natural Language Understanding System.

---

---

References

2023

• https://python.langchain.com/docs/modules/data_connection/document_transformers/
  • QUOTE: When you want to deal with long pieces of text, it is necessary to split up that text into chunks. As simple as this sounds, there is a lot of potential complexity here. Ideally, you want to keep the semantically related pieces of text together. What "semantically related" means could depend on the type of text. This notebook showcases several ways to do that.
  • At a high level, text splitters work as following:
    • Split the text up into small, semantically meaningful chunks (often sentences).
    • Start combining these small chunks into a larger chunk until you reach a certain size (as measured by some function).
    • Once you reach that size, make that chunk its own piece of text and then start creating a new chunk of text with some overlap (to keep context between chunks).
  • That means there are two different axes along which you can customize your text splitter:
    • How the text is split
    • How the chunk size is measured

---