SemEval-2014, Task 1
A SemEval-2014, Task 1 is a SemEval-2014 task that ...
- AKA: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.
- See: SemEval, Distributional Semantic Model.
References
2014
- http://alt.qcri.org/semeval2014/task1/
- QUOTE: Distributional Semantic Models (DSMs) approximate the meaning of words with vectors summarizing their patterns of co-occurrence in corpora. Recently, several compositional extensions of DSMs (Compositional DSMs, or CDSMs) have been proposed, with the purpose of representing the meaning of phrases and sentences by composing the distributional representations of the words they contain (e.g., [1], [2], [4], [5]). Despite the ever increasing interest in the field, the development of adequate benchmarks for CDSMs, especially at the sentence level, is still lagging behind. Existing data sets, such as those introduced by [3] and [2], are limited to a few hundred instances of very short sentences with a fixed structure. On the other hand, in the last ten years, several large data sets have been developed for various computational semantics tasks, such as Semantic Text Similarity (STS) or Recognizing Textual Entailment (RTE). Working with such data sets, however, requires dealing with issues, such as identifying multiword expressions, recognizing named entities or accessing encyclopedic knowledge, that are not what CDSMs are expected to handle. The latter should be evaluated on data sets involving difficulties associated to semantic compositionality (e.g., contextual synonymy and other lexical variation phenomena, active / passive and other syntactic alternations, impact of negation, determiners and other grammatical elements), that do not necessarily occur frequently in, e.g., the STS and RTE data sets.
With these considerations in mind, we developed SICK (Sentences Involving Compositional Knowledge), a data set aimed at filling the void, including a large number of sentence pairs that are rich in the lexical, syntactic and semantic phenomena that CDSMs are expected to account for, but do not require dealing with other aspects of existing sentential data sets (idiomatic multiword expressions, named entities, telegraphic language) that are not within the scope of compositional distributional semantics. Moreover, we distinguished between generic semantic knowledge about general concept categories (such as knowledge that a couple is formed by a bride and a groom) and encyclopedic knowledge about specific instances of concepts (e.g., knowing the fact that the current president of the US is Barack Obama). The SICK data set contains many examples of the former, but none of the latter.
- QUOTE: Distributional Semantic Models (DSMs) approximate the meaning of words with vectors summarizing their patterns of co-occurrence in corpora. Recently, several compositional extensions of DSMs (Compositional DSMs, or CDSMs) have been proposed, with the purpose of representing the meaning of phrases and sentences by composing the distributional representations of the words they contain (e.g., [1], [2], [4], [5]). Despite the ever increasing interest in the field, the development of adequate benchmarks for CDSMs, especially at the sentence level, is still lagging behind. Existing data sets, such as those introduced by [3] and [2], are limited to a few hundred instances of very short sentences with a fixed structure. On the other hand, in the last ten years, several large data sets have been developed for various computational semantics tasks, such as Semantic Text Similarity (STS) or Recognizing Textual Entailment (RTE). Working with such data sets, however, requires dealing with issues, such as identifying multiword expressions, recognizing named entities or accessing encyclopedic knowledge, that are not what CDSMs are expected to handle. The latter should be evaluated on data sets involving difficulties associated to semantic compositionality (e.g., contextual synonymy and other lexical variation phenomena, active / passive and other syntactic alternations, impact of negation, determiners and other grammatical elements), that do not necessarily occur frequently in, e.g., the STS and RTE data sets.