ACE-2003

See: ACE Benchmark Task, ACE Program, ACE-2004.

---

---

References

• http://www.nist.gov/speech/tests/ace/ace03/

• (Zhang et al., 2006)
  • "In the ACE 2003 data, the training set consists of 674 documents and 9683 relation instances while the test set consists of 97 documents and 1386 relation instances. The ACE 2003 data defines 5 entity types, 5 major relation types and 24 relation subtypes.

• (Kambhatla, 2004) ⇒ Nanda Kambhatla. (2004). Combining lexical, syntactic, and semantic features with maximum entropy models for extracting relations. Poster. In: Proceedings of [[ACL 2004]
  • "Automatic Content Extraction (ACE, 2004) is an evaluation conducted by NIST to measure Entity Detection and Tracking (EDT) and relation detection and characterization (RDC). The EDT task entails the detection of mentions of entities and chaining them together by identifying their coreference. In ACE vocabulary, entities are objects, mentions are references to them, and relations are explicitly or implicitly stated relationships among entities. Entities can be of five types: persons, organizations, locations, facilities, and geo-political entities (geographically defined regions that define a political boundary, e.g. countries, cities, etc.). Mentions have levels: they can be names, nominal expressions or pronouns.
  • "The RDC task detects implicit and explicit relations between entities identified by the EDT task. Explict relations occur in text with explicit evidence suggesting the relationship. Implicit relations need not have explicit supporting evidence in text, though they should be evident from a reading of the document.
  • "Here is an example:
    • “The American Medical Association voted yesterday to install the heir apparent as its president-elect, rejecting a strong, upstart challenge by a District doctor who argued that the nation’s largest physicians’ group needs stronger ethics and new leadership.
    • “In electing Thomas R. Reardon, an Oregon general practitioner who had been the chairman of its board, ...
  • "In this fragment, all the underlined phrases are mentions referring to the American Medical Association, or to Thomas R. Reardon or the board (an organization) of the American Medical Association. Moreover, there is an explicit management relation between chairman and board, which are references to Thomas R. Reardon and the board of the American Medical Association respectively. Relation extraction is hard, since successful extraction implies correctly detecting both the argument mentions, correctly chaining these mentions to their respective entities, and correctly determining the type of relation that holds between them.

• (Culotta and Sorensen, 2004) ⇒ Aron Culottaand J. S. Sorensen. (2004). “Dependency Tree Kernels for Relation Extraction.” In: Proceedings ofACL 2004.
  • "Although training was done over all 24 relation subtypes, we evaluate only over the 5 highlevel relation types. Thus, classifying a RESIDENCE relation as a LOCATED relation is deemed correct.
  • "Figure 3 - Distribution over relation types in training data. At_Located ~300, Role_Staff ~200, Role_Member ~200, Role_Mgmt ~170, Part_Part-of ~155, At_based-in ~85, At_residence ~70, Near_Relative_loc ~40, etc …

Reported Results

Paper	Method	5Mt-P	5Mt-R	5Mt-F	24St-P	24St-R	24St-F
ZZJZ07	Composite kernel (cntxt. sens.)	80.8	68.4	74.1	65.2	54.9	59.6
ZZJZ07	Conv. tree kernel (cntxt.sense.)	80.1	63.8	71.0	63.4	51.9	57.1
ZZSZ06	Composite Kernel 2 (poly exp)	77.3	65.6	70.9	64.9	51.2	57.2
ZZSZ06	Composite Kernel 2 (linear comb)	76.3	63.0	69.0
ZZS06	ConvTreeKernel(PT+EI+Sem feat.)	76.3	63.0	69.0	64.6	50.76	56.83
ZZS06	ConvTreeKernel(PT+Entity information(EI))	76.1	62.9	68.9
ZSZZ05	Feature-based SVM	77.2	60.7	68.0	63.1	49.5	55.5
HBM05	contig+bag-o-words kernel plus all features				72.2	44.5	55.1
ZZS06	ConvTreeKernel only (Parse tree)	72.8	53.8	61.9
N04	Feature-based MaxEnt				63.5	45.2	52.8
ZZSZ06	Entity kernel only (Parse tree)	79.5	34.6	48.2
CS04	Tree kernel	67.1	35.0	45.8
HBM05	contig+bag-o-words kernel non SRL features				60.5	20.3	30.4

• 5Mt ⇒ 5 Major types
• 24St ⇒ 24 Subtypes
• P ⇒ Precision.
• R ⇒ Recall.
• F ⇒ F-Measure.
• ZZJZ07 ⇒ (ZhouZJZ, 2007)
• ZZSZ06 ⇒ (ZhangZSZ, 2006)
• ZZS06 ⇒ (ZhangZS, 2006)
• ZSZZ05 ⇒ (Zhou et al., 2005)
• HBM05 ⇒ (HarabagiuBM, 2005)
• N04 ⇒ (Kambhatla, 2004)
• CS04 ⇒ (Culotta and Sorensen, 2004)