Marco-Elia-Nam (MEN) Semantic Relatedness Benchmark Task

A Marco-Elia-Nam (MEN) Semantic Relatedness Benchmark Task is a Semantic Relatedness Benchmark Task that evaluates the performance of semantic relatedness systems.

• Context:
  • Task Input: Linguistic Dataset.
  • Task Output: Semantic Relatedness Score between linguistic item-pairs.
  • Task Requirement(s):
    • Benchmark Datasets: MEN Semantic Relatedness Dataset,
    • Benchmark Performance Metrics: MEN Semantic Relatedness Score,
    • Benchmark Systems: TunedS, NaiveFL, NaiveSL, MixLDA.
• Example(s):
  • …
• Counter-Example(s):
  • Baroni-Lenci Evaluation of Semantic Similarity (BLESS) Benchmark Task.
• See: Semantic Similarity Measure, Word Relatedness Measure, Distributional Semantics, Multimodal Semantic Relatedness Model.

---

---

References

2014

• (Bruni et al., 2014) ⇒ Elia Bruni, Nam-Khanh Tran, and Marco Baroni. (2014). "Multimodal Distributional Semantics". In: Journal of Artificial Intelligence Research, 49.
  • QUOTE: The high-score MEN pairs include not only pairs of terms that are strictly taxonomically close (cathedral-church: 0.94) but also terms that are connected by broader semantic relations, such as whole-part (flower-petal: 0.92), item and related event (boat-fishing: 0.9),etc. For this reason, we prefer to refer to MEN as a semantic relatedness rather than similarity score data set. Note that WS is also capturing a broader notion of relatedness (Agirre et al., 2009). MEN is publicly available and it can be downloaded from: http://clic.cimec.unitn.it/~elia.bruni/MEN

---