Recall-Oriented Understudy for Gisting Evaluation (ROUGE)

Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics commonly used for text summarisation tasks. ROUGE scores assess the similarity and overlap between the words or phrases in the machine-generated text and the reference text.

$$ROUGE=\sum (Recallofn-grams)$$

The candidate text is the model output, and the reference text is the human written version.

3 Types of ROUGE

Type	What it measures	Used to evaluate
ROUGE-N	the overlap of n-grams, which are contiguous sequences of n words	grammatical correctness and fluency
ROUGE-L	the longest common subsequence (LCS) between the candidate and the reference	semantic similarity and content coverage
ROUGE-S	the skip-bigram overlap, where a skip-bigram is a bi-gram with at most 1 intervening word	coherence and local cohesion

Precision, recall and F1 score are computed for all 3 types, and only the unit being counted changes.

ROUGE-N

$$Recall=\frac{Overlappingnumberofn-grams}{Numberofn-gramsinreference}$$ $$Precision=\frac{Overlappingnumberofn-grams}{Numberofn-gramsincandidate}$$

ROUGE-L

$$Recall=\frac{NumberofwordsinLCS}{Numberofwordsinreference}$$ $$Precision=\frac{NumberofwordsinLCS}{Numberofwordsincandidate}$$

• a subsequence keeps the order of the words but allows gaps, so no fixed window is needed
• this rewards a summary that follows the reference in order, even when extra words sit in between

ROUGE-S

$$Recall=\frac{Numberofskip-bigrams}{Numberofskip-bigramsinreference}$$ $$Precision=\frac{Numberofskip-bigrams}{Numberofskip-bigramsincandidate}$$

Code

import evaluate
 
rouge = evaluate.load('rouge')
predictions = ["Transformers Transformers are fast plus efficient",
               "Good Morning", "I am waiting for new Transformers"]
references = [
    ["HuggingFace Transformers are fast efficient plus awesome",
     "Transformers are awesome because they are fast to execute"],
    ["Good Morning Transformers", "Morning Transformers"],
    ["People are eagerly waiting for new Transformer models",
     "People are very excited about new Transformers"]
]
results = rouge.compute(predictions=predictions, references=references)
print(results)

{'rouge1': 0.6659340659340659, 'rouge2': 0.45454545454545453,
 'rougeL': 0.6146520146520146, 'rougeLsum': 0.6146520146520146}

• the evaluate library from HuggingFace supplies the metric, and evaluate.load('rouge') fetches it
• each prediction is paired with a list of references, so several acceptable answers can be given
• rouge1 and rouge2 are ROUGE-N at n equal to 1 and 2, while rougeL is the longest common subsequence version

Comparison with BLEU

Point	ROUGE	BLEU
metric it uses	recall	precision
what it answers	how much of the reference was captured	how much of the output was correct
usual task	text summarisation	machine translation

Both count overlapping n-grams, and the difference is which side the denominator sits on. The precision oriented counterpart is Bilingual Evaluation Understudy (BLEU).