2014 OverviewoftheEnglishSlotFilling
- (Surdeanu & Ji, 2014) ⇒ Mihai Surdeanu, and Heng Ji. (2014). “Overview of the English Slot Filling Track at the TAC2014 Knowledge Base Population Evaluation.” In: Proceedings of the Text Analysis Conference (TAC 2014).
Subject Headings: Slot Filling Task, TAC-KBP 2014 Slot Filling Task.
Notes
Cited By
Quotes
Abstract
We overview the English Slot Filling (SF) track of the TAC2014 Knowledge Base Population (KBP) evaluation. The goal of this KBP track is to promote research in the extraction of binary relations between named and numeric entities from free text. The main changes this year include: (a) the inclusion of ambiguous queries, i.e., queries that point to multiple real-life entities with the same name; (b) accepting outputs created through inference; and (c) a simplification of the task and of the input format by removing references to the knowledge base for the entities included in queries. The SF track attracted 31 registered teams, out of which 18 teams submitted at least one run. The highest score this year was 36.72 F1, with a median of 19.80 F1.
1 Introduction
The Knowledge Base Population (KBP) track at TAC 2014 aims to promote research on automated systems that discover information about named entities and incorporate this information in a knowledge source, or database. This effort can be seen as a natural continuation of previous conferences and evaluations, such as the Message Understanding Conference (MUC) (Grishman and Sundheim, 1996) and the Automatic Content Extraction (ACE) evaluations [1]
…
7 Concluding Remarks
With respect to the SF task, this year’s evaluation continues the positive trends seen in the past year. First, SF continues to be popular, with 18 teams submitting results in 67 different runs (the largest number of runs to date). SF continues to attract new participants: out of the 18 participating teams, six were first participants this year. Second, this year’s results show increased performance. The maximum score of systems that participated in the past two SF evaluations increased from 33.89 F1 points in 2013 to 36.72 F1 this year. Similarly, the median score of all submissions increased from 15.7 F1 points to 19.8. This is despite the fact that the test queries this year were more complex, containing, at the same time, ambiguous entities (i.e., same name, multiple real-world entities), and obscure entities, with minimal support in the document collection.
While this improvement is very positive, it is important to note that SF systems are still far from human performance on this task. The top system this year achieves 52% of human performance, and the median system is at only 28% of human performance. We are still far from solving the SF problem. We believe it is important to continue this evaluation, to allow information extraction technology to advance and mature.
With respect to future work, one immediate change that is necessary is to update the reference knowledge base from the 2008_Wikipedia to a more recent and modern resource, such as DBpedia[2]. This will minimize the disconnect between the SF training data available to participants and the assessment of results, which uses the live Wikipedia. Furthermore, we would like to incorporate (require ?) more inference in the SF task (maybe through a closer interaction with Cold Start).
Footnotes
References
;
| Author | volume | Date Value | title | type | journal | titleUrl | doi | note | year | |
|---|---|---|---|---|---|---|---|---|---|---|
| 2014 OverviewoftheEnglishSlotFilling | Heng Ji Mihai Surdeanu | Overview of the English Slot Filling Track at the TAC2014 Knowledge Base Population Evaluation |