Long Context Language Model (LCLM)

A Long Context Language Model (LCLM) is an LLM whose context length is greater than 32,000.

• Example(s):
  • GPT-4 32K.
  • Claude 2 100K.
  • Llama 2.
  • ...
• See: Short-Context LLM.

---

---

References

2023

• (An et al., 2023) ⇒ Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. (2023). “L-Eval: Instituting Standardized Evaluation for Long Context Language Models.” In: arXiv preprint arXiv:2307.11088. doi:10.48550/arXiv.2307.11088
  • QUOTE: ... It also remains unclear whether developing these long context models can offer substantial gains on practical downstream tasks over retrieval-based methods or models simply trained on chunked contexts. To address this challenge, we propose to institute standardized evaluation for long context language models. Concretely, we develop L-Eval which contains 411 long documents and over 2,000 query-response pairs manually annotated and checked by the authors encompassing areas such as law, finance, school lectures, lengthy conversations, news, long-form novels, and meetings. L-Eval also adopts diverse evaluation methods and instruction styles, enabling a more reliable assessment of Long Context Language Models (LCLMs). ...

---