A RAG answer is judged by whether the evidence supports it, not by whether it looks like a reference answer.
Why BLEU and ROUGE fall short
Bilingual Evaluation Understudy (BLEU) and ROUGE count shared word sequences (n-grams) with a reference answer. They measure similarity of wording, not correctness or evidence.
- a wrong answer can score high, for example "within five business days" against a reference that says two differs by one word
- a correct answer can score low if it is paraphrased or adds a helpful explanation
- neither one checks whether the answer is supported by the retrieved passages
RAG triad
Three questions, where each one points to a different component to fix:
| Dimension | Question it answers | Component to fix |
|---|---|---|
| Context relevance | Are the retrieved passages relevant to the question | Retriever |
| Faithfulness | Is every claim in the answer supported by the passages | Generator |
| Answer relevance | Does the answer address the question that was asked | Prompt or generator |
Faithfulness
A claim is one atomic statement of fact. LLM-as-judge tools such as RAGAS split the answer into claims and check each one against the passages. Their verdicts should be calibrated against human labels.
Faithful is not the same as true
A true claim that the passages do not support still counts as unfaithful, because in an audit it cannot be defended.
Worked example
The passage says: "Cash deposits above 20,000 dollars require enhanced review by the compliance team within two business days." The answer says: "Deposits over 20,000 dollars trigger an enhanced review by compliance within two business days, and failures are reported to the police."
- claim 1, deposits over 20,000 dollars trigger enhanced review: supported
- claim 2, the review is done by compliance: supported
- claim 3, it happens within two business days: supported
- claim 4, failures are reported to the police: not in the passage
Faithfulness is . Claim 4 sounds plausible and might even be true, but nothing retrieved supports it, so it counts as a hallucination.
Code
A transparent stand-in for an LLM judge: a claim counts as supported if at least 60% of its content words appear in the passage.
import re
STOP = set("a an the is are be by of to in on for and or with within must that this it as at from".split())
def tokenize(text):
return re.findall(r"[a-z0-9§\-]+", text.lower())
def content_words(text):
"""Tokens that carry meaning, with stop words removed."""
return {t for t in tokenize(text) if t not in STOP}
def faithfulness(claims, passage, threshold=0.6):
"""Share of claims whose content words are mostly found in the passage."""
passage_words = content_words(passage)
results = []
for claim in claims:
words = content_words(claim)
support = len(words & passage_words) / max(len(words), 1)
results.append((claim, round(support, 2), support >= threshold))
score = sum(r[2] for r in results) / len(results)
return score, results
passage = "Clause CP-12 §4.3: cash deposits above 20,000 dollars require enhanced review by the compliance team within two business days."
claims = [
"deposits over 20,000 dollars trigger enhanced review",
"the review is done by compliance",
"it happens within two business days",
"failures are reported to the police",
]
score, results = faithfulness(claims, passage)
print(score) # 0.75Notes:
- the answer must be split into atomic claims before scoring, which an LLM judge does automatically
- word overlap cannot tell "within two business days" from "within five business days" if the words appear in the passage, and it misses paraphrased support, which is why LLM-as-judge methods exist
Faithfulness checks the answer, and the next question is whether to retrieve at all, see Long Context and Lost in the Middle.