Bilingual Evaluation Understudy (BLEU)

Bilingual Evaluation Understudy (BLEU) evaluates the similarity between a target sentence and a generated sentence. It counts the number of n-grams that appear in both the generated sentence and the target sentence, and it is mostly used in machine translation applications.

common n-gram sizes used are 1, 2, 3 or 4
BLEU calculates the precision of matching n-grams
the target sentence is the human reference, and the predicted sentence is the model output

Precision

P r ec i s i o n = \frac{T o t a l n u mb er o f cor r ec tl y p r e d i c t e d n - g r am s in t h e t a r g e t se n t e n ce}{T o t a l n u mb er o f n - g r am s in t h e p r e d i c t e d se n t e n ce}

Working with 1-grams, which are single words:

Predicted Sentence: He eats an apple
Target Sentence: He ate an apple
the precision is $\frac{3}{4}$ , because "eats" has no match

Clipped Precision

Plain precision can be gamed by repetition. A predicted sentence of "He He He" scores well because every word appears in the target, even though the sentence is useless.

To avoid repetitions, clipped precision is used instead of precision:

each word from the predicted sentence is compared with all the target sentences, and if the word has a match in any target sentence it is considered correct
the count for each correct word is limited to the maximum number of times that word occurs across all target sentences

For a predicted sentence of "He He He eats tasty fruit" against 2 target sentences, the clipped precision is 2/6, because the repeated instances of "He" are not counted.

Geometric Average Precision

The precisions of the separate n-gram sizes are combined into 1 number:

G eo m e t r i c A v er a g e P r ec i s i o n (N) = exp (n = 1 \sum N w_{n} lo g p_{n}) = n = 1 \prod N p_{n}^{w_{n}}

For the default of 4 n-gram sizes with equal weights:

= (p_{1})^{1/4} \cdot (p_{2})^{1/4} \cdot (p_{3})^{1/4} \cdot (p_{4})^{1/4}

$p_{n}$ is the clipped precision for that n-gram size
$w_{n}$ is the weight given to that n-gram size
a geometric average is used, so a score of 0 at any n-gram size drives the whole product to 0

Brevity Penalty

A brevity penalty is then added to penalise sentences that are too short, since shorter sentences can generate a misleading probability.

B r e v i t y P e na l t y = {1, e^{1 - \frac{r}{c}}, i f c > r i f c \leq r

Where $c$ is the predicted length and $r$ is the target length.

a prediction longer than the target is not penalised, because precision already punishes extra words
a prediction shorter than the target is scaled down, and the shorter it is the harder the penalty

Scoring

B L E U = B r e v i t y P e na l t y \cdot G eo m e t r i c A v er a g e P r ec i s i o n (N)

Code

import nltk
from nltk import word_tokenize
from nltk.translate.bleu_score import SmoothingFunction
 
ref = 'The guard arrived late because it was raining.'
cand = 'The guard arrived late because of the rain.'
smoothie = SmoothingFunction().method1
reference = word_tokenize(ref)
candidate = word_tokenize(cand)
weights = (0.25, 0.25, 0.25, 0.25)
BLEUscore = nltk.translate.bleu_score.sentence_bleu([reference], candidate, weights, smoothing_function=smoothie)
print(BLEUscore)

the reference is passed inside a list, because a sentence can be scored against several target sentences at once
weights of 4 values at 0.25 each is the default 4-gram setting, and it matches the geometric average above
weights here shows 0.25 for 1-gram precision, 2-gram precision, 3-gram precision, and 4-gram precision
All 4 grams' scores receives equal importance to how much they affect the final BLEU score, and the weights add up to 1
a SmoothingFunction avoids a score of 0 when a higher order n-gram has no match at all

BLEU is precision oriented, and its recall oriented counterpart is Recall-Oriented Understudy for Gisting Evaluation (ROUGE).

JJ

Explorer