LLM-based Natural Language Processing (NLP) System

An LLM-based Natural Language Processing (NLP) System is an NLP system that is an LLM-based system (which critically relies on a large language model).

• Example(s):
  • an LLM-based Commercial Contract Categorizer.
  • an LLM-based Q/A System.
  • an LLM-based Information Extraction System.
  • ...
• See: LLM-Supported Clinical Trial System.

---

---

References

2023

• chat
  • Q: Create a wiki table with 3 columns: "NLP Task Type", "Specific Benchmark Tasks", and "Pure LLM Solution?". In the first column, list 14 NLP task types. In the second column, list some specific benchmark tasks typically used to evaluate performance on each task type. In the third column, indicate whether a pure LLM can typically solve the task or if additional training and optimization may be required. Mark up each technical term or noun phrase using wiki link double square bracket annotation. Reply in a formatted text box.
  • A:

NLP Task Type	Specific Benchmark Tasks	Pure LLM Solution?
Named Entity Recognition (NER)	CoNLL-2003 NER, OntoNotes 5.0	No, additional training and optimization required
Sentiment Analysis	Stanford Sentiment Treebank (SST), IMDb movie review dataset	No, additional training and optimization required
Machine Translation	WMT (Workshop on Machine Translation), IWSLT (International Workshop on Spoken Language Translation)	No, additional training and optimization required
Text Summarization	CNN/Daily Mail dataset, Gigaword dataset	No, additional training and optimization required
Question Answering	SQuAD (Stanford Question Answering Dataset), Natural Questions (NQ)	No, additional training and optimization required
Part-of-Speech Tagging	Penn Treebank dataset, Universal Dependencies (UD) dataset	No, additional training and optimization required
Text Classification	AG News, Yelp Review dataset	No, additional training and optimization required
Coreference Resolution	CoNLL-2012 Shared Task, GAP (Gendered Ambiguous Pronouns)	No, additional training and optimization required
Semantic Parsing	GeoQuery, WikiSQL dataset	No, additional training and optimization required
Relation Extraction	SemEval Task 8, TACRED (TAC Relation Extraction Dataset)	No, additional training and optimization required
Dependency Parsing	Penn Treebank dataset, Universal Dependencies (UD) dataset	No, additional training and optimization required
Topic Modeling	20 Newsgroups dataset, Reuters-21578 dataset	No, additional training and optimization required
Language Modeling	Penn Treebank dataset, WikiText-2, WikiText-103	No, additional training and optimization required
Paraphrase Identification	Microsoft Research Paraphrase Corpus (MRPC), Quora Question Pairs dataset	No, additional training and optimization required

2022

• (Wei, Tay et al., 2022) ⇒ Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.. (2022). “Emergent Abilities of Large Language Models.” In: Transactions on Machine Learning Research, 08/2022 (TMLR).
• (Srivastava et al., 2022) ⇒ Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and others. (2022). “Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models.” In: arXiv preprint arXiv:2206.04615.

---