BERTScore Evaluation Metric

A BERTScore Evaluation Metric is a NLG metric based on contextual similarity.

• Context:
  • It can utilize BERT (Model)'s pre-trained Contextual Embeddings for word comparisons.
  • It computes Cosine Similarity between words in a candidate sentence and words in a reference sentence.
  • It can be more effective than traditional Exact Match-based evaluation metrics, like BLEU Score.
  • It is particularly useful in tasks like Machine Translation and Image Captioning.
  • ...
• Example(s):
  • ...
• Counter-Example(s):
  • Word-Based Evaluation Metric, such as:
  • Human Tech Generation Evaluation, which relies on subjective assessments.
  • ...
• See: Natural Language Processing, Text Similarity Measure, Machine Translation Evaluation, Contextual Language Model.

---

---

References

2023

• GBard
  • BERTScore is an evaluation metric for text generation that uses the contextual embeddings from BERT to compute similarity between candidate and reference texts. It was first proposed in the paper BERTScore: Evaluating Text Generation with BERT: https://arxiv.org/abs/1904.09675 by Zhang et al. (2020).
  • BERTScore is a more comprehensive evaluation metric than traditional metrics like BLEU, which only consider exact matches between candidate and reference texts. BERTScore is able to capture more nuanced similarities between texts, such as semantic similarity and syntactic similarity. This makes it a more reliable metric for evaluating the quality of text generation.
  • Here are some of the advantages of using BERTScore:
    • It is more comprehensive than traditional metrics like BLEU.
    • It is able to capture more nuanced similarities between texts.
    • It is reliable and consistent.
    • It is easy to use.
  • Here are some of the disadvantages of using BERTScore:
    • It is computationally expensive.
    • It requires a large amount of data to train.
    • It is not always clear how to interpret the results.

2021

• (Hanna & Bojar, 2021) ⇒ Michael Hanna, and Ondřej Bojar. (2021). “A Fine-grained Analysis of BERTScore.” In: Proceedings of the Sixth Conference on Machine Translation.
  • ABSTRACT: BERTScore, a recently proposed automatic metric for machine translation quality, uses BERT, a large pre-trained language model to evaluate candidate translations with respect to a gold translation. Taking advantage of BERT’s semantic and syntactic abilities, BERTScore seeks to avoid the flaws of earlier approaches like BLEU, instead scoring candidate translations based on their semantic similarity to the gold sentence. However, BERT is not infallible; while its performance on NLP tasks set a new state of the art in general, studies of specific syntactic and semantic phenomena have shown where BERT’s performance deviates from that of humans more generally. This naturally raises the questions we address in this paper: what are the strengths and weaknesses of BERTScore? Do they relate to known weaknesses on the part of BERT? We find that while BERTScore can detect when a candidate differs from a reference in important content words, it is less sensitive to smaller errors, especially if the candidate is lexically or stylistically similar to the reference.

2019

• (Zhang et al., 2019) ⇒ Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. (2019). “Bertscore: Evaluating Text Generation with Bert.” arXiv preprint arXiv:1904.09675.
  • ABSTRACT: We propose BERTScore, an automatic evaluation metric for text generation. Analogously to common metrics, BERTScore computes a similarity score for each token in the candidate sentence with each token in the reference sentence. However, instead of exact matches, we compute token similarity using contextual embeddings. We evaluate using the outputs of 363 machine translation and image captioning systems. BERTScore correlates better with human judgments and provides stronger model selection performance than existing metrics. Finally, we use an adversarial paraphrase detection task to show that BERTScore is more robust to challenging examples when compared to existing metrics.

---