2004 InfExtrForQASyntacticPatterns

• (Jijkoun et al., 2004) ⇒ Valentin Jijkoun, Jori Mur, Maarten de Rijke. (2004). “Information Extraction for Question Answering: Improving recall through syntactic patterns.” In: Proceedings of COLING-2004.

Subject Headings: Pattern-based Relation Mention Recognition, Minipar, Question Answering.

Quotes

Abstract

• We investigate the impact of the precision/recall trade-off of information extraction on the performance of an offline corpus-based question answering (QA) system. One of our findings is that, because of the robust final answer selection mechanism of the QA system, recall is more important. We show that the recall of the extraction component can be improved using syntactic parsing instead of more common surface text patterns, substantially increasing the number of factoid questions answered by the QA system.

1 Introduction

• In our experiments we tried to understand whether linguistically involved methods such as parsing can be beneficial for information extraction, where rather shallow techniques are traditionally employed, and whether the abstraction from surface to syntactic structure of the text does indeed help to find more information, at the same time avoiding the time-consuming manual development of increasing numbers of surface patterns.

2 RelatedWork

• There is a large body of work on extracting semantic information using lexical patterns.

Hearst (1992) explored the use of lexical patterns for extracting hyponym relations, with patterns such as “such as.” Berland and Charniak (1999) extract “part-of” relations. Mann (2002) describes a method for extracting instances from text by means of part-of-speech patterns involving proper nouns.

• The use of lexical patterns to identify answers in corpus-based QA received lots of attention after a team taking part in one of the earlier QA Tracks at TREC showed that the approach was competitive at that stage (Soubbotin and Soubbotin, 2002; Ravichandran and Hovy, 2002). Different aspects of pattern-based methods have been investigated since.

E.g., Ravichandran et al. (2003). collect surface patterns automatically in an unsupervised fashion using a collection of trivia question and answer pairs as seeds. These patterns are then used to generate and assess answer candidates for a statistical QA system. Fleischman et al. (2003). focus on the precision of the information extracted using simple partof-speech patterns. They describe a machine learning method for removing noise in the collected data and showed that the QA system based on this approach outperforms an earlier state-of-the-art system. Similarly, Bernardi et al. (2003). combine the extraction of surface text patterns with WordNet based filtering of name-apposition pairs to increase precision, but found that it hurt recall more than it helped precision, resulting in fewer questions answered correctly when the extracted information is deployed for QA.

• The application of deeper NLP methods has also received much attention in the QA community.

The open-domain QA system by LCC (Moldovan et al., 2002) uses predicate-argument relations and lexical chaining to actually prove that a text snippet provides an answer to a question. Katz and Lin (2003) use syntactic dependency parsing to extract relations between words, and use these relations rather than individual words to retrieve sentences relevant to a question. They report a substantial improvement for certain types of questions for which the usual term-based retrieval performs quite poorly, but argue that deeper text analysis methods should be applied with care.

4.1 Extraction with Surface Text Patterns

• To extract information about roles, we used the set of surface patterns originally developed for the QA system we used at TREC 2003 (Jijkoun et al., 2004). The patterns are listed in Table 1. In these patterns, person is a phrase that is tagged as person by the Named Entity tagger, role is a word from a list of roles extracted from the WordNet (all hyponyms of the word ‘person,’ 15703 entries),1 role-verb is from a manually constructed list of “important” verbs (discovered, invented, etc.; 48 entries), leader is a phrase identifying leadership from a manually created list of leaders (president, minister, etc.; 22 entries). Finally, superlat is the superlative form of an adjective and location is a phrase tagged as location by the Named Entity tagger.
• Table 1 - Surface Patterns
  • Pattern Example
  • … role, person The British actress, Emma Thompson
  • … (superlat|first|last)..., person The first man to set foot on the moon, Armstrong
  • person,... role... Audrey Hepburn, goodwill ambassador for UNICEF.
  • person,... (superlat|first|last)... Brown, Democrats’ first black chairman.
  • person,... role-verb... Christopher Columbus, who discovered America,

... role person District Attoney Gil Garcetti

• • role... person The captain of the Titanic Edward John Smith
  • person,... leader... location Tony Blair, the prime minister of England
  • location... leader, person The British foreign secretary, Jack Straw

4.2 Extraction with Syntactic Patterns

• To use the syntactic structure of sentences for role information extraction, the collections were parsed with Minipar (Lin, 1998), a broad coverage dependency parser for English. Minipar is reported to achieve 88% precision and 80% recall with respect to dependency relations when evaluated on the SUSANNE corpus. We found that it performed well on the newpaper and newswire texts of our collections and was fairly robust to fragmented and not

well-formed sentences frequent in this domain. Before extraction, Minipar’s output was cleaned and made more compact. For example, we removed some empty nodes in the dependency parse to resolve non-local dependencies. While not loosing any important information, this made parses easier to analyse when developing patterns for extraction.

• Table 2 lists the patterns that were used to extract information about persons; we show syntactic dependencies as arrows from dependents to heads, with Minipar’s dependency labels above the arrows.
• As with the earlier surface patterns, role is one of the nouns in the list of roles (hyponyms of in WordNet), role-verb is one of the “important verbs.” The only restriction for person was that it should contain a proper noun.
• Table 2: Syntactic patterns
  • Pattern Example
  • Apposition person appo −−−!role a major developer, Joseph Beard
  • Apposition person appo −−−role Jerry Lewis, a Republican congressman
  • Clause person subj −−−!role-verb Bell invented the telephone
  • Person person person −−−!role Vice President Al Gore
  • Nominal modifier person nn −−−role businessman Bill Shockley
  • Subject person subj −−−!role Alvarado was chancellor from 1983 to 1984
  • Conjunction person conj −−−role Fu Wanzhong, director of the Provincial Department of Foreign Trade (this is a frequent parsing error)

References

• 1. Matthew Berland, Eugene Charniak, Finding parts in very large corpora, Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics on Computational Linguistics, p.57-64, June 20-26, 1999, College Park, Maryland doi:10.3115/1034678.1034697
• 2. R. Bernardi, V. Jijkoun, G. Mishne, and M. de Rijke. (2003). Selectively using linguistic resources throughout the question answering pipeline. In: Proceedings of the 2nd CoLogNET-ElsNET Symposium.
• 3. Michael Fleischman, Eduard Hovy, Abdessamad Echihabi, Offline strategies for online question answering: answering questions before they are asked, Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, p.1-7, July 07-12, 2003, Sapporo, Japan doi:10.3115/1075096.1075097
• 4. Marti A. Hearst, Automatic acquisition of hyponyms from large text corpora, Proceedings of the 14th conference on Computational linguistics, August 23-28, 1992, Nantes, France doi:10.3115/992133.992154
• 5. V. Jijkoun, G. Mishne, and M. de Rijke. (2003). Preprocessing Documents to Answer Dutch Questions. In: Proceedings of the 15th Belgian-Dutch Conference on Artificial Intelligence (BNAIC'03).
• 6. V. Jijkoun, G. Mishne, C. Monz, M. de Rijke, S. Schlobach, and O. Tsur. (2004). The University of Amsterdam at the TREC 2003 Question Answering Track. In: Proceedings of the TREC-2003 Conference.
• 7. B. Katz and J. Lin. (2003). Selectively using relations to improve precision in question answering. In: Proceedings of the EACL-2003 Workshop on Natural Language Processing for Question Answering.
• 8. Dekang Lin. (1998). Dependency-based evaluation of Minipar. In: Proceedings of the Workshop on the Evaluation of Parsing Systems.
• 9. Gideon S. Mann, Fine-grained proper noun ontologies for question answering, COLING-02 on SEMANET: building and using semantic networks, p.1-7, September 01, 2002 doi:10.3115/1118735.1118746
• 10. Dan Moldovan, Sanda M. Harabagiu, R. Girju, P. Morarescu, A. Novischi F. Lacatusu, A. Badulescu, and O. Bolohan. (2002). LCC tools for question answering. In: Proceedings of the TREC-2002.
• 11. TnT Statistical Part of Speech Tagging. (2003). URL: http://www.coli.uni-sb.de/~thorsten/tnt/.
• 12. CoNLL: Conference on Natural Language Learning. (2003). URL: http://cnts.uia.ac.be/signll/shared.html.
• 13. Deepak Ravichandran, Eduard Hovy, Learning surface text patterns for a Question Answering system, Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, July 07-12, 2002, Philadelphia, Pennsylvania doi:10.3115/1073083.1073092
• 14. Deepak Ravichandran, Abraham Ittycheriah, Salim Roukos, Automatic derivation of surface text patterns for a maximum entropy based question answering system, Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume of the Proceedings of HLT-NAACL 2003--short papers, p.85-87, May 27-June 01, 2003, Edmonton, Canada doi:10.3115/1073483.1073512
• 15. M. M. Soubbotin and S. M. Soubbotin. (2002). Use of patterns for detection of likely answer strings: A systematic approach. In: Proceedings of the TREC-2002 Conference.

---

,

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2004 InfExtrForQASyntacticPatterns	Valentin Jijkoun Maarten de Rijke Jori Mur			Information Extraction for Question Answering: Improving recall through syntactic patterns			http://acl.ldc.upenn.edu/C/C04/C04-1188.pdf