Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics commonly used for text summarisation tasks. ROUGE scores assess the similarity and overlap between the words or phrases in the machine-generated text and the reference text.

The candidate text is the model output, and the reference text is the human written version.

3 Types of ROUGE

TypeWhat it measuresUsed to evaluate
ROUGE-Nthe overlap of n-grams, which are contiguous sequences of n wordsgrammatical correctness and fluency
ROUGE-Lthe longest common subsequence (LCS) between the candidate and the referencesemantic similarity and content coverage
ROUGE-Sthe skip-bigram overlap, where a skip-bigram is a bi-gram with at most 1 intervening wordcoherence and local cohesion

Precision, recall and F1 score are computed for all 3 types, and only the unit being counted changes.

ROUGE-N

ROUGE-L

  • a subsequence keeps the order of the words but allows gaps, so no fixed window is needed
  • this rewards a summary that follows the reference in order, even when extra words sit in between

Example of LCS:

  • reference: "police killed the gunman"
  • candidate A: "police kill the gunman"
  • candidate B: "the gunman kill police"
CandidateLCSLCS lengthRecallPrecisionF1
Apolice the gunman33/43/40.75
Bthe gunman22/42/40.5
  • in candidate A, "kill" does not match "killed", so the LCS skips that word and joins "police" to "the gunman" across the gap
  • in candidate B, "police" comes at the end, so it is out of order and cannot join the LCS
  • ROUGE-2 gives both candidates the same score, because each one matches only the bigram "the gunman"
  • ROUGE-L gives candidate A the higher score, because candidate A keeps more of the reference word order

Remember

A subsequence keeps the order and allows gaps. A substring keeps the order and allows no gaps.

ROUGE-S

Skip-bigrams of the reference "police killed the gunman":

  • pairs with no gap: "police killed", "killed the", "the gunman"
  • pairs with a gap of 1 word: "police the", "killed gunman"

This gives 5 skip-bigrams for the reference.

Example of skip-bigrams:

CandidateMatched skip-bigramsRecallPrecisionF1
A: "police kill the gunman""the gunman", "police the"2/52/50.4
B: "the gunman kill police""the gunman"1/51/50.2
  • ROUGE-2 gives A and B the same score, because each one matches only "the gunman"
  • ROUGE-S gives A the higher score, because "police the" is a correct pair in the correct order with "kill" in the gap
  • ROUGE-S is more flexible than ROUGE-2 because a gap is permitted, but less flexible than ROUGE-L because the gap has a size limit

No ROUGE-S in evaluate

The evaluate library returns only rouge1, rouge2, rougeL and rougeLsum, so it does not give a ROUGE-S score.

Code

import evaluate
 
rouge = evaluate.load('rouge')
predictions = ["Transformers Transformers are fast plus efficient",
               "Good Morning", "I am waiting for new Transformers"]
references = [
    ["HuggingFace Transformers are fast efficient plus awesome",
     "Transformers are awesome because they are fast to execute"],
    ["Good Morning Transformers", "Morning Transformers"],
    ["People are eagerly waiting for new Transformer models",
     "People are very excited about new Transformers"]
]
results = rouge.compute(predictions=predictions, references=references)
print(results)
{'rouge1': 0.6659340659340659, 'rouge2': 0.45454545454545453,
 'rougeL': 0.6146520146520146, 'rougeLsum': 0.6146520146520146}
  • the evaluate library from HuggingFace supplies the metric, and evaluate.load('rouge') fetches it
  • each prediction is paired with a list of references, so several acceptable answers can be given
  • rouge1 and rouge2 are ROUGE-N at n equal to 1 and 2, while rougeL is the longest common subsequence version

Comparison with BLEU

PointROUGEBLEU
metric it usesrecallprecision
what it answershow much of the reference was capturedhow much of the output was correct
usual tasktext summarisationmachine translation

Both count overlapping n-grams, and the difference is which side the denominator sits on. The precision oriented counterpart is Bilingual Evaluation Understudy (BLEU).