Meteor Universal Scoring Task

A Meteor Universal Scoring Task is a Scoring Task that evaluates translation hypothesis-reference pairs via alignment and by calculating sentence-level similarity score.

• AKA: Meteor Scoring Task.
• Context:
  • Task Input(s): paraphrase tables and function word lists extracts from the bitext used to train MT systems.
  • Task Output(s): Meteor Score.
  • It can be solved by Meteor Universal Scoring System that implements Meteor Universal Scoring Algorithms.
• Example(s):
  • http://www.cs.cmu.edu/~alavie/METEOR/examples.html
  • …
• Counter-Example(s):
  • Winograd Schema Challenge Benchmark Task,
  • ROUGE Benchmark Task,
  • BLEU Benchmark Task.
• See: 2014 ACL Workshop on Statistical Machine Translation, Statistical Machine Translation, Performance Metric, Similarity Score, WordNet Database, Sockeye Neural Machine Translation Toolkit.

---

---

References

2014

• (Denkowski & Lavie, 2014) ⇒ Michael J. Denkowski, and Alon Lavie. (2014). “Meteor Universal: Language Specific Translation Evaluation for Any Target Language". In: Proceedings of the Ninth Workshop on Statistical Machine Translation (WMT@ACL 2014). DOI:10.3115/v1/W14-3348.
  • QUOTE: Meteor evaluates translation hypotheses by aligning them to reference translations and calculating sentence-level similarity scores. For a hypothesis-reference pair, the space of possible alignments is constructed by exhaustively identifying all possible matches between the sentences according to the following matchers:
    • Exact: Match words if their surface forms are identical.
    • Stem: Stem words using a language appropriate Snowball Stemmer (...) and match if the stems are identical.
    • Synonym: Match words if they share membership in any synonym set according to the WordNet database (...).
    • Paraphrase: Match phrases if they are listed as paraphrases in a language appropriate paraphrase table (...).

---