2017 NewsQAAMachineComprehensionData
- (Trischler et al., 2017) ⇒ Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. (2017). “NewsQA: A Machine Comprehension Dataset.” In: Proceedings of the 2nd Workshop on Representation Learning for NLP (Rep4NLP@ACL 2017).
Subject Headings: Question-Answer Dataset; NewsQA Dataset; Reading Comprehension Dataset, FastQA Dataset.
Notes
Cited By
- Google Scholar: ~ 352 Citations, Retrieved: 2020-12-27.
Quotes
Abstract
We present NewsQA, a challenging machine comprehension dataset of over 100, 000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10, 000 news articles from CNN, with answers consisting of spans of text in the articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. Analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (13.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available online.
1. Introduction
(...)
(...)
In this paper, we present a challenging new large-scale dataset for machine comprehension: NewsQA. It contains 119,633 natural language questions posed by crowdworkers on 12,744 news articles from CNN. In SQuAD, crowdworkers are tasked with both asking and answering questions given a paragraph. In contrast, NewsQA was built using a collection process designed to encourage exploratory, curiosity-based questions that may better reflect realistic information-seeking behaviors. Particularly, a set of crowdworkers were tasked to answer questions given a summary of the article, i.e. the CNN article highlights. A separate set of crowdworkers selects answers given the full article, which consist of word spans in the corresponding articles. This gives rise to interesting patterns such as questions that may not be answerable by the original article.
(...)
NewsQA is closely related to the SQuAD dataset: it is crowdsourced, with answers given by spans of text within an article rather than single words or entities, and there are no candidate answers from which to choose. The challenging characteristics of NewsQA that distinguish it from SQuAD are as follows:
- 1. Articles in NewsQA are significantly longer (6x on average) and come from a distinct domain.
- 2. Our collection process encourages lexical and syntactic divergence between questions and answers.
- 3. A greater proportion of questions requires reasoning beyond simple word- and context-matching.
- 4. A significant proportion of questions have no answer in the corresponding article.
2. Related Datasets
3. Collection Methodology
4. Data Analysis
5. Baseline Model
6. Experiments
7. Conclusion
Appendices
A. Implementation Details
References
BibTeX
@inproceedings{2017_NewsQAAMachineComprehensionData,
author = {Adam Trischler and
Tong Wang and
Xingdi Yuan and
Justin Harris and
Alessandro Sordoni and
Philip Bachman and
Kaheer Suleman},
editor = {Phil Blunsom and
Antoine Bordes and
Kyunghyun Cho and
Shay B. Cohen and
Chris Dyer and
Edward Grefenstette and
Karl Moritz Hermann and
Laura Rimell and
Jason Weston and
Scott Yih},
title = {NewsQA: A Machine Comprehension Dataset},
booktitle = {Proceedings of the 2nd Workshop on Representation Learning for NLP
(Rep4NLP@ACL 2017)},
address = {Vancouver, Canada},
month = {August},
pages = {191--200},
publisher = {Association for Computational Linguistics},
year = {2017},
url = {https://doi.org/10.18653/v1/w17-2623},
doi = {10.18653/v1/w17-2623},
}