2018 TheNarrativeQAReadingComprehens

• (Kocisky et al., 2018) ⇒ Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. (2018). “The NarrativeQA Reading Comprehension Challenge.” In: Trans. Assoc. Comput. Linguistics, 6.

Subject Headings: Reading Comprehension Dataset; NarrativeQA Dataset; NarrativeQA Reading Comprehension Challenge.

Notes

• Link(s):
  • MIT Press: https://www.mitpressjournals.org/doi/pdfplus/10.1162/tacl_a_00023
  • arXiv: https://arxiv.org/abs/1712.07040
  • Transactions of the Association for Computational Linguistics: https://transacl.org/ojs/index.php/tacl/article/view/1197

Cited By

• Google Scholar: ~ 202 Citations, Retrieved: 2020-12-13.

Quotes

Abstract

Reading comprehension (RC) - in contrast to information retrieval - requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.

2. Review of Reading Comprehension Datasets and Models

Table 1: Comparison of datasets.

Dataset	Documents	Questions	Answers
MCTest (Richardson et al., 2013)	660 short stories, grade school level	2640 human generated, based on the document	multiple choice
CNN/Daily Mail (Hermann et al., 2015)	93K+220K news articles	387K+997K Cloze-form, based on highlights	entities
Children’s Book Test (CBT) (Hill et al., 2016)	687K of 20 sentence passages from 108 children’s books	Cloze-form, from the 21st sentence	multiple choice
BookTest (Bajgar et al., 2016)	14.2M, similar to CBT	Cloze-form, similar to CBT	multiple choice
SQuAD (Rajpurkar et al., 2016)	23K paragraphs from 536 Wikipedia articles	108K human generated, based on the paragraphs	spans
NewsQA (Trischler et al., 2016)	13K news articles from the CNN dataset	120K human generated, based on headline, highlights	spans
MS MARCO (Nguyen et al., 2016)	1M passages from 200K+ documents retrieved using the queries	100K search queries	human generated, based on the passages
SearchQA (Dunn et al., 2017)	6.9m passages retrieved from a search engine using the queries	140k human generated Jeopardy! questions	human generated Jeopardy! answers
NarrativeQA (this paper)	1,572 stories (books, movie scripts) & human generated summaries	46,765 human generated, based on summaries	human generated, based on summaries

References

2017a

• (Dunn et al., 2017) ⇒ Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Guney, Volkan Cirik, and Kyunghyun Cho. (2017). “SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine.” In: ePrint: abs/1704.05179.

2017b

• (Trischler et al., 2017) ⇒ Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. (2017). “NewsQA: A Machine Comprehension Dataset.” In: Proceedings of the 2nd Workshop on Representation Learning for NLP (Rep4NLP@ACL 2017).

2016a

• (Bajgar et al., 2016) ⇒ Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. (2016). “Embracing Data Abundance: BookTest Dataset for Reading Comprehension.” In: ePrint: abs/1610.00956.

2016b

• (Hill et al., 2016) ⇒ Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. (2016). “The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations.” In: Proceedings of the 4th International Conference on Learning Representations (ICLR 2016) Conference Track.

2016c

• (Nguyen et al., 2016) ⇒ Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. (2016). “MS MARCO: A Human Generated MAchine Reading COmprehension Dataset.” In: Proceedings of the Workshop on Cognitive Computation: Integrating neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Information Processing Systems (NIPS 2016).

2016d

• (Rajpurkar et al., 2016) ⇒ Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. (2016). “SQuAD: 100,000+ Questions for Machine Comprehension of Text.” In: arXiv preprint arXiv:1606.05250

2015

• (Hermann et al., 2015) ⇒ Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. (2015). “Teaching Machines to Read and Comprehend.” In: Proceedings of the 28th International Conference on Neural Information Processing Systems (NIPS'15). ePrint: arXiv:1506.03340v3

2013

• (Richardson et al., 2013) ⇒ Matthew Richardson, Christopher J. C. Burges, and Erin Renshaw. (2013). “MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text.” In: Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP 2013). A meeting of SIGDAT, a Special Interest Group of the ACL.

BibTeX

@article{2018_TheNarrativeQAReadingComprehens,
  author    = {Tomas Kocisky and
               Jonathan Schwarz and
               Phil Blunsom and
               Chris Dyer and
               Karl Moritz Hermann and
               Gabor Melis and
               Edward Grefenstette},
  title     = {The NarrativeQA Reading Comprehension Challenge},
  journal   = {Trans. Assoc. Comput. Linguistics},
  volume    = {6},
  pages     = {317--328},
  year      = {2018},
  url       = {https://transacl.org/ojs/index.php/tacl/article/view/1197},
}

---

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2018 TheNarrativeQAReadingComprehens	Chris Dyer Edward Grefenstette Karl Moritz Hermann Phil Blunsom Gabor Melis Tomas Kocisky Jonathan Schwarz			The NarrativeQA Reading Comprehension Challenge						2018