2022 TrainingComputeOptimalLargeLang
- (Hoffmann et al., 2022) ⇒ Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. (2022). “Training Compute-optimal Large Language Models.” In: Advances in Neural Information Processing Systems, 35. doi:10.48550/arXiv.2203.15556
Subject Headings: LLM Scaling Laws, Chinchilla LLM.
Notes
- It suggests more complex scaling relationships between:
- LLM Model scaling: Increasing model size shows diminishing returns and performance saturation. Over 100B parameters, further scale provides limited gains for a fixed dataset.
- LLM Dataset scaling: Dataset size improvements also show diminishing benefits. Returns plateau around 1B tokens for a 100B+ parameter model.
- LLM Compute scaling: Optimal configurations balance model width, depth, batch size, and memory bandwidth depending on hardware.
- It explores the interplay between model size, training data size, and computing when training large language models to find the most efficient balance. The optimal scaling depends on the compute budget.
- It recommends increasing model size & data size together but stopping training before full convergence. The goal is optimal efficiency within constraints, not maximizing performance.
- It presents Chinchilla - a 70B parameter model trained on 4x the data of the 280B Gopher. With comparable computing to Gopher, Chinchilla showed gains on some but not all benchmarks. Multiple factors contributed including the increased data.
- It finds empirical scaling laws that provide guidance on hyperparameter choices but notes these may evolve as hardware and methodology advances. More research is required to further understand the complex relationships between model scale, data scale and model quality across different tasks.
Cited By
Quotes
Abstract
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled. We test this hypothesis by training a predicted compute-optimal model, Chinchilla, that uses the same compute budget as Gopher but with 70B parameters and 4× more more data. Chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B) on a large range of downstream evaluation tasks. This also means that Chinchilla uses substantially less compute for fine-tuning and inference, greatly facilitating downstream usage. As a highlight, Chinchilla reaches a state-of-the-art average accuracy of 67.5% on the MMLU benchmark, greater than a 7% improvement over Gopher.
References
;