2020 ScalingLawsforNeuralLanguageMod
- (Kaplan et al., 2020) ⇒ Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. (2020). “Scaling Laws for Neural Language Models.” In: arXiv preprint arXiv:2001.08361.
Subject Headings: Scaling Laws in Deep Learning, Transformer Model Optimization, Compute-Efficient Training Strategies, Language Model Generalization, Sample Efficiency in Large Language Models.
Notes
- The paper identifies power-law relationships between model size, dataset size, and compute, showing that performance improves predictably with scale across these factors.
- The paper demonstrates that model architecture, such as depth or width, has a minimal effect on performance compared to scaling parameters like model size, data, and compute.
- The paper reveals that larger models are more sample-efficient, achieving similar or better performance with fewer data and training steps than smaller models.
- The paper suggests that optimal training efficiency is achieved by training large models on modest datasets and halting training well before full convergence.
- The paper finds that overfitting can be predicted and mitigated by maintaining a balance between model size and dataset size, using a simple ratio to avoid diminishing returns.
- The paper emphasizes that training curves follow predictable patterns, enabling researchers to forecast final performance based on early training data.
- The paper highlights that larger models, when trained with the appropriate compute budget, can be significantly more compute-efficient than smaller models trained to convergence.
- The paper shows that performance scales smoothly across multiple orders of magnitude, with no significant deviation in trends, even as model size increases dramatically.
- The paper advocates for the use of larger batch sizes for training large models, with the ideal batch size scaling with the gradient noise of the model.
- The paper concludes that scaling model size is more impactful than increasing data size or training time, recommending a focus on larger models for future improvements.
- The paper predicts that as compute budgets increase, larger models will continue to outperform smaller ones, suggesting that "big models may be more important than big data."
Cited By
Quotes
Abstract
We study empirical scaling laws for language model performance on the cross-entropy loss. The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude. Other architectural details such as network width or depth have minimal effects within a wide range. Simple equations govern the dependence of overfitting on model/dataset size and the dependence of training speed on model size. These relationships allow us to determine the optimal allocation of a fixed compute budget. Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence.
Table of Contents
Introduction Background and Methods Empirical Results and Basic Power Laws Charting the Infinite Data Limit and Overfitting Scaling Laws with Model Size and Training Time Optimal Allocation of the Compute Budget Related Work Discussion Appendices A. Summary of Power Laws B. Empirical Model of Compute-Efficient Frontier C. Caveats D. Supplemental Figures .
References
;