Metric for Evaluation of Translation with Explicit Ordering (METEOR)

Metric for Evaluation of Translation with Explicit Ordering (METEOR) is used to assess the quality of machine translation systems. It complements other popular metrics, and its distinctive feature is that it takes the sequence of words in the output sentence into account.

Counting overlapping words alone treats a scrambled sentence as good as an ordered one. METEOR considers the importance of word order in evaluating the translation quality.

Chunk Penalty

To account for word order, a chunk penalty is included in the calculation of the METEOR metric.

Intuitively it represents the idea that a good translation should not only have words that are synonymous with the reference, but the words should also be in the correct order and grouped together in meaningful chunks.

$$p=0.5{\left(\frac{c}{u_m}\right)}^3$$

Where:

• $c$ is the number of chunks in the candidate
• $u_m$ is the number of unigrams in the candidate

A chunk is a run of words (group of words) that matches the reference in order. Few chunks means the matched words came out in long ordered runs, and many chunks means the same words came out scattered.

Scoring

$$M=F_{mean}(1-p)$$

Where $F_{mean}$ is a modified F1 score specifically used in METEOR.

• the matching part of the score is the $F_{mean}$ term
• the ordering part of the score is the $(1-p)$ term, which scales the result down as the output becomes more scattered

Code

from nltk.translate import meteor
from nltk import word_tokenize
 
score = round(meteor([word_tokenize('The cat sat on the mat')],
                     word_tokenize('The cat was sat on the mat')), 4)
print('The METEOR score is: ' + str(score))

The METEOR score is: 0.9654

• meteor([list of reference tokens], candidate tokens)
• from the above, we can see that the reference is passed inside a list and the candidate is passed on its own, so the argument order is reference first
• both inputs must be tokenized, since meteor works on lists of words rather than raw strings
• the score stays high here because the extra word "was" breaks the match into only a few chunks
• round(..., 4) here keeps 4 decimal places

METEOR is described as complementing Bilingual Evaluation Understudy (BLEU), which scores n-gram precision without considering order.