Position Independent Word Error Rate (PER) Measure

A Position Independent Word Error Rate (PER) Measure is a Performance Metric that compares the words in the two sentences without taking the word order into account.

• Context:
  • It can be used to benchmarking NLP systems such as:
    • Speech Recognition Systems,
    • Word Embedding Systems,
    • Neural Machine Translation Systems,
    • OOV Detection Systems.
• Example(s):
  • $\mathrm{PER}=\displaystyle \dfrac{1}{N_{\text {ref }}^{*}} \sum_{k=1}^{K} \min _{r} d_{\mathrm{PER}}\left(r e f_{k, r}, h y p_{k}\right)$ as described in Popovic & Ney (2007),
  • …
• Counter-Example(s):
  • Word Error Rate,
  • F-Measure,
  • METEOR Metric,
  • ROUGE Metric,
  • NIST Score,
  • Word Accuracy (WAcc) Measure,
  • Phonetic Levenshtein Distance.
  • ASR Latency.
  • BLEU Score.
• See: Recall, Information Retrieval, Speech Recognition, Machine Translation, Levenshtein Distance, Phoneme.

---

---

References

2007

• (Popovic & Ney, 2007) ⇒ Maja Popovic, and Hermann Ney. (2007). “Word Error Rates: Decomposition over POS Classes and Applications for Error Analysis.” In: Proceedings of the Second Workshop on Statistical Machine Translation (WMT@ACL 2007).
  • QUOTE: The word error rate (WER) is based on the Levenshtein distance (Levenshtein, 1966) - the minimum number of substitutions, deletions and insertions that have to be performed to convert the generated text $hyp$ into the reference text $ref$. A shortcoming of the WER is the fact that it does not allow reorderings of words, whereas the word order of the hypothesis can be different from word order of the reference even though it is correct translation. In order to overcome this problem, the position independent word error rate (PER) compares the words in the two sentences without taking the word order into account. The PER is always lower than or equal to the WER. On the other hand, shortcoming of the PER is the fact that the word order can be important in some cases. Therefore the best solution is to calculate both word error rates.
    (...)
    Calculation of PER: The PER can be calculated using the counts $n\left(e, hyp_k\right)$ and $n\left(e, ref_{k,r}\right)$ of a word $e$ in the hypothesis sentence $hyp_k$ and the reference sentence $ref_{k,r}$ respectively:

$\mathrm{PER}=\displaystyle \dfrac{1}{N_{\text {ref }}^{*}} \sum_{k=1}^{K} \min _{r} d_{\mathrm{PER}}\left(r e f_{k, r}, h y p_{k}\right)$

(1)

$d_{\mathrm{PER}}\left(ref_{k, r}, hyp_{k}\right) = \displaystyle \dfrac{1}{2}\left(\left|N_{r e f_{k, r}}-N_{h y p_{k}}\right|+\sum_{e}\left|n\left(e, r e f_{k, r}\right)-n\left(e, hyp_{k}\right)\right|\right)$

---