BLOOM Large Language Model (LLM)

Context:
- …
Example(s):
- …
Counter-Example(s):
- GPT-3, GPT-4,
- Transformer-XL.
- T5 LLM,
- LlAMA LLM.
See: Transformer (Machine Learning Model), Large Language Model, Hugging Face, Natural Language Processing (NLP), Deep Learning, Language Generation.

References

https://huggingface.co/bigscience/bloom
- QUOTE: ... BLOOM is an autoregressive Large Language Model (LLM), trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources. As such, it is able to output coherent text in 46 languages and 13 programming languages that is hardly distinguishable from text written by humans. BLOOM can also be instructed to perform text tasks it hasn't been explicitly trained for, by casting them as text generation tasks. ...

(Wikipedia, 2023) ⇒ https://en.wikipedia.org/wiki/BLOOM_(language_model) Retrieved:2023-5-7.
- BigScience Large Open-science Open-access Multilingual Language Model (BLOOM) is a transformer-based large language model. It was created by over 1000 AI researchers to provide a free large language model for everyone who wants to try. Trained on around 366 billion tokens over March through July 2022, it is considered an alternative to OpenAI's GPT-3 with its 176 billion parameters. BLOOM uses a decoder-only transformer model architecture modified from Megatron-LM GPT-2. The BLOOM project was started by a co-founder of Hugging Face. Six main groups of people were involved, including HuggingFace's BigScience team, the Microsoft DeepSpeed team, the NVIDIA Megatron-LM team, the IDRIS/GENCI team, the PyTorch team, and the volunteers in the BigScience Engineering workgroup.BLOOM was trained using data of 46 natural languages and 13 programming languages. In total, 1.6 TeraByte pre-processed text was converted into 350 billion unique tokens as BLOOM's training datasets.

(Scaoe et al., 2022) ⇒ Teven L. Scaoe, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné et al. (2022). “Bloom: A 176b-parameter Open-access Multilingual Language Model.” arXiv preprint arXiv:2211.05100
- ABSTRACT: Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.

List of large language models
Name	Release dateTemplate:Efn	Developer	Number of parametersTemplate:Efn	Corpus size	LicenseTemplate:Efn	Notes
BLOOM	July 2022	Large collaboration led by Hugging Face	175 billion	350 billion tokens (1.6TB)^[1]	Responsible AI	Essentially GPT-3 but trained on a multi-lingual corpus (30% English excluding programming languages)