NarrativeQA Dataset

A NarrativeQA Dataset is a QA dataset that consists of over 1,500 stories from books and movie scripts as well as human generated summaries.

• Context:
  • Benchmark Task: NarrativeQA Reading Comprehension Challenge.
• Example(s):
  • …
• Counter-Example(s):
  • a CoQA Dataset,
  • a MS COCO Dataset,
  • a NewsQA Dataset,
  • a RACE Dataset,
  • a SearchQA Dataset,
  • a SQuAD Dataset.
• See: Question-Answering System, Natural Language Processing Task, Natural Language Understanding Task, Natural Language Generation Task.

---

---

References

2018

• (Kocisky et al., 2018) ⇒ Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. (2018). “The NarrativeQA Reading Comprehension Challenge.” In: Trans. Assoc. Comput. Linguistics, 6.
  • QUOTE: To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience.
    (...)

Table 1: Comparison of datasets.

Dataset	Documents	Questions	Answers
MCTest (Richardson et al., 2013)	660 short stories, grade school level	2640 human generated, based on the document	multiple choice
CNN/Daily Mail (Hermann et al., 2015)	93K+220K news articles	387K+997K Cloze-form, based on highlights	entities
Children’s Book Test (CBT) (Hill et al., 2016)	687K of 20 sentence passages from 108 children’s books	Cloze-form, from the 21st sentence	multiple choice
BookTest (Bajgar et al., 2016)	14.2M, similar to CBT	Cloze-form, similar to CBT	multiple choice
SQuAD (Rajpurkar et al., 2016)	23K paragraphs from 536 Wikipedia articles	108K human generated, based on the paragraphs	spans
NewsQA (Trischler et al., 2017)	13K news articles from the CNN dataset	120K human generated, based on headline, highlights	spans
MS MARCO (Nguyen et al., 2016)	1M passages from 200K+ documents retrieved using the queries	100K search queries	human generated, based on the passages
SearchQA (Dunn et al., 2017)	6.9m passages retrieved from a search engine using the queries	140k human generated Jeopardy! questions	human generated Jeopardy! answers
NarrativeQA (this paper)	1,572 stories (books, movie scripts) & human generated summaries	46,765 human generated, based on summaries	human generated, based on summaries

---