2019 RoBERTaARobustlyOptimizedBERTPr

• (Liu, Ott et al., 2019) ⇒ Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. (2019). “RoBERTa: A Robustly Optimized BERT Pretraining Approach.” In: CoRR, abs/1907.11692.

Subject Headings: RoBERTa System, RoBERTa Model, BERT.

Notes

• MS Academic: https://academic.microsoft.com/paper/2965373594/reference
• DBLP: abs-1907-11692
• ArXiv: 1907.11692

Cited By

• Google Scholar: ~ 1,188 Citations (Retrieved:2021-06-14).
• Semantic Scholar: ~ 3,371 Citations (Retrieved:2021-06-14).
• MS Academic: ~ 5,500 Citations (Retrieved:2021-10-14).

Quotes

Abstract

Language model pretraining has led to significant performance gains but careful comparison between different approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes, and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results highlight the importance of previously overlooked design choices, and raise questions about the source of recently reported improvements. We release our models and code.

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2019 RoBERTaARobustlyOptimizedBERTPr	Omer Levy Luke Zettlemoyer Danqi Chen Veselin Stoyanov Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Mike Lewis			RoBERTa: {A} Robustly Optimized {BERT} Pretraining Approach						2019