Semantic Word Similarity Benchmark Task

A Semantic Word Similarity Benchmark Task is a Semantic Similarity Benchmark Task that evaluates the performance of semantic word similarity systems.

• Context:
  • Task Input: natural language data.
  • Task Output: semantic word similarity score.
  • Task Requirement(s):
    • Benchmark datasets: semantic word similarity datasets.
    • ML training system or training corpora.
    • Benchmark performance metrics: semantic word similarity measure.
    • Semantic word similarity systems (a baseline system + participating systems).
  • It ranges from being a Monolingual Semantic Word Similarity Benchmark Task, to being a Multilingual Semantic Word Similarity Benchmark Task, to being a Cross-lingual Semantic Word Similarity Benchmark Task.
• Example(s):
  • a SemEval Benchmark Task such as:
    • SemEval-2017 Task 1,
    • SemEval-2017 Task 2.
  • …
• Counter-Example(s):
  • a Gene Semantic Similarity Benchmark Task,
  • a Semantic Relation Mention Recognition Benchmark Task.
  • a Named Entity Recognition Benchmark Task;
  • a Coreference Resolution Benchmark Task;
  • a Question Answering Benchmark Task;
  • a Document Summarization Benchmark Task.
• See: Word Similarity, Semantic Similarity Analysis Task, Natural Language Processing Task, Semantic Analysis Task, Synthetic Hierarchy Learning.

---

---

References

2015

• (Vilnis & McCallum, 2015) ⇒ Luke Vilnis, and Andrew McCallum. (2015). “Word Representations via Gaussian Embedding.” In: arXiv preprint arXiv:1412.6623 submitted to ICRL 2015.
  • QUOTE: We demonstrated the effectiveness of these embeddings on a linguistic task requiring asymmetric comparisons, as well as standard word similarity benchmarks, learning of synthetic hierarchies, and several qualitative examinations.

---