2021 SwitchTransformersScalingtoTril

(Fedus et al., 2021) ⇒ William Fedus, Barret Zoph, and Noam Shazeer. (2021). “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” In: The Journal of Machine Learning Research, 23(1). DOI:10.5555/3586589.3586709.

Subject Headings: Switch Transformer Network, Transformer Network, Mixture-of-Experts (MoE), MoE Routing Algorithm.

Notes

Online Resource(s):
- Source code: https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/transformer/moe.py

Cited By

Google Scholar: ~ 78+ Citations.

Quotes

Abstract

In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication costs and training instability -- we address these with the Switch Transformer. We simplify the MoE routing algorithm and design intuitive improved models with reduced communication and computational costs. Our proposed training techniques help wrangle the instabilities and Fe show large sparse models may be trained, for the first time, with lower precision (bfloat16) formats. We design models based off T5-Base and T5-Large to obtain up to 7x increases in pre-training speed with the same computational resources. These improvements extend into multilingual settings where we measure gains over the mT5-Base version across all 101 languages. Finally, we advance the current scale of language models by pre-training up to trillion parameter models on the “Colossal Clean Crawled Corpus” and achieve a 4x speedup over the T5-XXL model.

References

BibTeX

@article{2021_SwitchTransformersScalingtoTril,
  author    = {William Fedus and
               Barret Zoph and
               Noam Shazeer},
  title     = {Switch Transformers: Scaling to Trillion Parameter Models with Simple
               and Efficient Sparsity},
  journal   = {CoRR},
  volume    = {abs/2101.03961},
  year      = {2021},
  url       = {https://arxiv.org/abs/2101.03961},
  archivePrefix = {arXiv},
  eprint    = {2101.03961},
}

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2021 SwitchTransformersScalingtoTril	Barret Zoph Noam Shazeer William Fedus			Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity						2021