2016 TheLAMBADADatasetWordPrediction

(Paperno et al., 2016) ⇒ Denis Paperno, German Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. (2016). “The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context.” In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016) Volume 1: Long Papers.

Subject Headings: Lambada Dataset, GPT-2 Benchmark Task, Word Prediction Task.

Notes

Cited By

Google Scholar: ~ 89 Citations, Retrieved:2021-01-03.

Quotes

Abstract

We introduce LAMBADA, a dataset to evaluate the capabilities of computational models for text understanding by means of a word prediction task. LAMBADA is a collection of narrative passages sharing the characteristic that human subjects are able to guess their last word if they are exposed to the whole passage, but not if they only see the last sentence preceding the target word. To succeed on LAMBADA, computational models cannot simply rely on local context, but must be able to keep track of information in the broader discourse. We show that LAMBADA exemplifies a wide range of linguistic phenomena, and that none of several state-of-the-art language models reaches accuracy above 1% on this novel benchmark. We thus propose LAMBADA as a challenging test set, meant to encourage the development of new models capable of genuine understanding of broad context in natural language text.

References

BibTeX

@inproceedings{2016_TheLAMBADADatasetWordPrediction,
  author    = {Denis Paperno and
               German Kruszewski and
               Angeliki Lazaridou and
               Quan Ngoc Pham and
               Raffaella Bernardi and
               Sandro Pezzelle and
               Marco Baroni and
               Gemma Boleda and
               Raquel Fernandez},
  title     = {The LAMBADA dataset: Word prediction requiring a broad discourse
               context},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational
               Linguistics (ACL 2016) Volume 1: Long Papers},
  publisher = {The Association for Computer Linguistics},
  year      = {2016},
  url       = {https://doi.org/10.18653/v1/p16-1144},
  doi       = {10.18653/v1/p16-1144},
}

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2016 TheLAMBADADatasetWordPrediction	Raffaella Bernardi Marco Baroni Denis Paperno German Kruszewski Angeliki Lazaridou Quan Ngoc Pham Sandro Pezzelle Gemma Boleda Raquel Fernandez			The LAMBADA Dataset: Word Prediction Requiring a Broad Discourse Context						2016