2018 KnowWhatYouDontKnowUnanswerable

• (Rajpurkar et al., 2018) ⇒ Pranav Rajpurkar, Robin Jia, and Percy Liang. (2018). “Know What You Don't Know: Unanswerable Questions for SQuAD.” In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018), Volume 2: Short Papers.

Subject Headings: SQuAD Dataset; SQuADRUn Dataset; Machine Reading Comprehension System; SQuAD Benchmark Task.

Cited By

• Google Scholar: ~ 679 Citations, Retrieved: 2020-12-12.

Quotes

Abstract

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuADRUn, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50, 000 unanswerable questions written adversarially by crowd-workers to look similar to answerable ones. To do well on SQuADRUn, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuADRUn is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQuADRUn. We release SQuADRUn to the community as the successor to SQuAD.

Introduction

(...)

In this work, we construct SQuADRUn, a new dataset that combines the existing questions in SQuAD with 53,775 new, unanswerable questions about the same paragraphs. Crowdworkers crafted these questions so that (1) they are relevant to the paragraph, and (2) the paragraph contains a plausible answer—something of the same type as what the question asks for. Two such examples are shown in Figure 1.

Figure 1: Two unanswerable questions written by crowdworkers, along with plausible (but incorrect) answers. Relevant keywords are shown in blue.

Article: Endangered Species Act Paragraph: “ . . . Other legislation followed, including the Migratory Bird Conservation Act of 1929, a 1937 treaty prohibiting the hunting of right and gray whales, and the Bald Eagle Protection Act of 1940. These later laws had a low cost to society—the species were relatively rare—and little opposition was raised.”

Question 1: “Which laws faced significant opposition?” Plausible Answer: later laws

Question 2: “What was the name of the 1937 treaty?” Plausible Answer: Bald Eagle Protection Act

(...)

BibTeX

@inproceedings{2018_KnowWhatYouDontKnowUnanswerable,
  author    = {Pranav Rajpurkar and
               Robin Jia and
               Percy Liang},
  editor    = {Iryna Gurevych and
               Yusuke Miyao},
  title     = {Know What You Don't Know: Unanswerable Questions for SQuAD},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational
               Linguistics (ACL 2018), Volume 2: Short Papers},
  address   = {Melbourne, Australia},
  date      = {July 15-20, 2018},
  pages     = {784--789},
  publisher = {Association for Computational Linguistics},
  year      = {2018},
  url       = {https://www.aclweb.org/anthology/P18-2124/},
  doi       = {10.18653/v1/P18-2124},
}

---

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2018 KnowWhatYouDontKnowUnanswerable	Percy Liang Pranav Rajpurkar Robin Jia			Know What You Don't Know: Unanswerable Questions for SQuAD						2018