Long Context Language Model (LCLM)
Jump to navigation
Jump to search
A Long Context Language Model (LCLM) is an LLM whose context length is greater than 32,000.
- Example(s):
- GPT-4 32K.
- Claude 2 100K.
- Llama 2.
- ...
- See: Short-Context LLM.
References
2023
- (An et al., 2023) ⇒ Chenxin An, Shansan Gong, Ming Zhong, Mukai Li, Jun Zhang, Lingpeng Kong, and Xipeng Qiu. (2023). “L-Eval: Instituting Standardized Evaluation for Long Context Language Models.” In: arXiv preprint arXiv:2307.11088. doi:10.48550/arXiv.2307.11088
- QUOTE: ... It also remains unclear whether developing these long context models can offer substantial gains on practical downstream tasks over retrieval-based methods or models simply trained on chunked contexts. To address this challenge, we propose to institute standardized evaluation for long context language models. Concretely, we develop L-Eval which contains 411 long documents and over 2,000 query-response pairs manually annotated and checked by the authors encompassing areas such as law, finance, school lectures, lengthy conversations, news, long-form novels, and meetings. L-Eval also adopts diverse evaluation methods and instruction styles, enabling a more reliable assessment of Long Context Language Models (LCLMs). ...