Pretrained Large Neural Language Model (Pretrained LLM)

A Pretrained Large Neural Language Model (Pretrained LLM) is a pretrained language model that is a large language model.

• Context:
  • It can be an input to a In-Context Learning System.
  • It can be an input to a LLM Fine-Tuning System.
  • ...
  • It can range from being a Pure Pretrained LLM to being a Finetuned LLM (such as an instruction-tuned LLM).
  • ...
• Example(s):
  • a General Purpose Pretrained LLM, such as GPT-4.
  • a Domain-Specific Pretrained LLM, such as:
    • a Pretrained Biomedical LLM (e.g. BioGPT) or a Pretrained Protein LLM.
    • a Pretrained Software LLM, such as Codex LLM.
    • a Pretrained Finance LLM, such as Bloomberg LLM.
    • a Pretrained Legal LLM, such as [[]].
  • a Proprietary Pretrained LLM, such as:
    • a Google Pretrained LLM, Azure Pretrained LLM, ...
  • …
• Counter-Example(s):
  • a Pre-Trained Small Language Model.
  • a Pre-Trained Image Generation Model.
• See: Language Model Metamodel, LLM Architecture, ULMFiT.

---

---

References

2023

• (Wikipedia, 2023) ⇒ https://en.wikipedia.org/wiki/Large_language_model#List_of_large_language_models Retrieved:2023-3-19.

List of large language models

Name	Release dateTemplate:Efn	Developer	Number of parametersTemplate:Efn	Corpus size	LicenseTemplate:Efn	Notes
BERT	2018	Google	340 million	3.3 billion words	Apache 2.0	early and influential language model
GPT-2	2019	OpenAI	1.5 billion	40GB (~10 billion tokens)	MIT	general-purpose model based on transformer architecture
GPT-3	2020	OpenAI	175 billion	499 billion tokens	Template:Public web API	A fine-tuned variant of GPT-3, termed GPT-3.5, was made available to the public through a web interface called ChatGPT in 2022.
GPT-Neo	March 2021	EleutherAI	2.7 billion	825 GiB	MIT	The first of a series of free GPT-3 alternatives released by EleutherAI. GPT-Neo outperformed an equivalent-size GPT-3 model on some benchmarks, but was significantly worse than the largest GPT-3.
GPT-J	June 2021	EleutherAI	6 billion	825 GiB	Apache 2.0	GPT-3-style language model
Ernie 3.0 Titan	December 2021	Baidu	260 billion	4 Tb	Proprietary	Chinese-language LLM. Ernie Bot is based on this model.
Claude	December 2021	Anthropic	52 billion	400 billion tokens	Template:Closed beta	fine-tuned for desirable behavior in conversations
GLaM (Generalist Language Model)	December 2021	Google	1.2 trillion	1.6 trillion tokens	Proprietary	sparse mixture-of-experts model, making it more expensive to train but cheaper to run inference compared to GPT-3
LaMDA (Language Models for Dialog Applications)	January 2022	Google	137 billion	1.56T words	Proprietary	specialized for response generation in conversations
Megatron-Turing NLG	October 2021	Microsoft and Nvidia	530 billion	338.6 billion tokens	Restricted web access	standard architecture but trained on a supercomputing cluster
GPT-NeoX	February 2022	EleutherAI	20 billion	825 GiB	Apache 2.0	based on the Megatron architecture
Chinchilla	March 2022	DeepMind	70 billion	1.3 trillion tokens	Proprietary	reduced-parameter model trained on more data
PaLM (Pathways Language Model)	April 2022	Google	540 billion	768 billion tokens	Proprietary	aimed to reach the practical limits of model scale
OPT (Open Pretrained Transformer)	May 2022	Meta	175 billion	180 billion tokens	Template:Non-commercial researchTemplate:Efn	GPT-3 architecture with some adaptations from Megatron
YaLM 100B	June 2022	Yandex	100 billion	1.7TB	Apache 2.0	English-Russian model
BLOOM	July 2022	Large collaboration led by Hugging Face	175 billion	350 billion tokens (1.6TB)	Responsible AI	Essentially GPT-3 but trained on a multi-lingual corpus (30% English excluding programming languages)
AlexaTM (Teacher Models)	November 2022	Amazon	20 billion	1.3 trillion	Template:Public web API	bidirectional sequence-to-sequence architecture
LLaMA (Large Language Model Meta AI)	February 2023	Meta	65 billion	1.4 trillion	Template:Non-commercial researchTemplate:Efn	trained on a large 20-language corpus to aim for better performance with fewer parameters.
GPT-4	March 2023	OpenAI	UnknownTemplate:Efn	Unknown	Template:Public web API	Available for ChatGPT Plus users. Microsoft confirmed that GPT-4 model is used in Bing Chat.

2023

• (Zhao, Zhou et al., 2023) ⇒ Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. (2023). “A Survey of Large Language Models.” In: arXiv preprint arXiv:2303.18223. doi:10.48550/arXiv.2303.18223

2022

• (Li, Tang et al., 2021) ⇒ Junyi Li, Tianyi Tang, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. (2021). “Pretrained Language Models for Text Generation: A Survey.” arXiv:2105.10311 https://doi.org/10.48550/arXiv.2201.05273
  • ABSTRACT: Text Generation aims to produce plausible and readable text in a human language from input data. The resurgence of deep learning has greatly advanced this field, in particular, with the help of neural generation models based on pre-trained language models (PLMs). Text generation based on PLMs is viewed as a promising approach in both academia and industry. In this paper, we provide a survey on the utilization of PLMs in text generation. We begin with introducing three key aspects of applying PLMs to text generation: 1) how to encode the input into representations preserving input semantics which can be fused into PLMs; 2) how to design an effective PLM to serve as the generation model; and 3) how to effectively optimize PLMs given the reference text and to ensure that the generated texts satisfy special text properties. Then, we show the major challenges arisen in these aspects, as well as possible solutions for them. We also include a summary of various useful resources and typical text generation applications based on PLMs. Finally, we highlight the future research directions which will further improve these PLMs for text generation. This comprehensive survey is intended to help researchers interested in text generation problems to learn the core concepts, the main techniques and the latest developments in this area based on PLMs.

---