Recall-Oriented Understudy for Gisting Evaluation (ROUGE) is a set of metrics commonly used for text summarisation tasks. ROUGE scores assess the similarity and overlap between the words or phrases in the machine-generated text and the reference text.
The candidate text is the model output, and the reference text is the human written version.
3 Types of ROUGE
| Type | What it measures | Used to evaluate |
|---|---|---|
| ROUGE-N | the overlap of n-grams, which are contiguous sequences of n words | grammatical correctness and fluency |
| ROUGE-L | the longest common subsequence (LCS) between the candidate and the reference | semantic similarity and content coverage |
| ROUGE-S | the skip-bigram overlap, where a skip-bigram is a bi-gram with at most 1 intervening word | coherence and local cohesion |
Precision, recall and F1 score are computed for all 3 types, and only the unit being counted changes.
ROUGE-N
ROUGE-L
- a subsequence keeps the order of the words but allows gaps, so no fixed window is needed
- this rewards a summary that follows the reference in order, even when extra words sit in between
Example of LCS:
- reference: "police killed the gunman"
- candidate A: "police kill the gunman"
- candidate B: "the gunman kill police"
| Candidate | LCS | LCS length | Recall | Precision | F1 |
|---|---|---|---|---|---|
| A | police the gunman | 3 | 3/4 | 3/4 | 0.75 |
| B | the gunman | 2 | 2/4 | 2/4 | 0.5 |
- in candidate A, "kill" does not match "killed", so the LCS skips that word and joins "police" to "the gunman" across the gap
- in candidate B, "police" comes at the end, so it is out of order and cannot join the LCS
- ROUGE-2 gives both candidates the same score, because each one matches only the bigram "the gunman"
- ROUGE-L gives candidate A the higher score, because candidate A keeps more of the reference word order
Remember
A subsequence keeps the order and allows gaps. A substring keeps the order and allows no gaps.
ROUGE-S
Skip-bigrams of the reference "police killed the gunman":
- pairs with no gap: "police killed", "killed the", "the gunman"
- pairs with a gap of 1 word: "police the", "killed gunman"
This gives 5 skip-bigrams for the reference.
Example of skip-bigrams:
| Candidate | Matched skip-bigrams | Recall | Precision | F1 |
|---|---|---|---|---|
| A: "police kill the gunman" | "the gunman", "police the" | 2/5 | 2/5 | 0.4 |
| B: "the gunman kill police" | "the gunman" | 1/5 | 1/5 | 0.2 |
- ROUGE-2 gives A and B the same score, because each one matches only "the gunman"
- ROUGE-S gives A the higher score, because "police the" is a correct pair in the correct order with "kill" in the gap
- ROUGE-S is more flexible than ROUGE-2 because a gap is permitted, but less flexible than ROUGE-L because the gap has a size limit
No ROUGE-S in
evaluateThe
evaluatelibrary returns onlyrouge1,rouge2,rougeLandrougeLsum, so it does not give a ROUGE-S score.
Code
import evaluate
rouge = evaluate.load('rouge')
predictions = ["Transformers Transformers are fast plus efficient",
"Good Morning", "I am waiting for new Transformers"]
references = [
["HuggingFace Transformers are fast efficient plus awesome",
"Transformers are awesome because they are fast to execute"],
["Good Morning Transformers", "Morning Transformers"],
["People are eagerly waiting for new Transformer models",
"People are very excited about new Transformers"]
]
results = rouge.compute(predictions=predictions, references=references)
print(results){'rouge1': 0.6659340659340659, 'rouge2': 0.45454545454545453,
'rougeL': 0.6146520146520146, 'rougeLsum': 0.6146520146520146}
- the
evaluatelibrary from HuggingFace supplies the metric, andevaluate.load('rouge')fetches it - each prediction is paired with a list of references, so several acceptable answers can be given
rouge1androuge2are ROUGE-N at n equal to 1 and 2, whilerougeLis the longest common subsequence version
Comparison with BLEU
| Point | ROUGE | BLEU |
|---|---|---|
| metric it uses | recall | precision |
| what it answers | how much of the reference was captured | how much of the output was correct |
| usual task | text summarisation | machine translation |
Both count overlapping n-grams, and the difference is which side the denominator sits on. The precision oriented counterpart is Bilingual Evaluation Understudy (BLEU).