A RAG answer is judged by whether the evidence supports it, not by whether it looks like a reference answer.

Why BLEU and ROUGE fall short

Bilingual Evaluation Understudy (BLEU) and ROUGE count shared word sequences (n-grams) with a reference answer. They measure similarity of wording, not correctness or evidence.

  • a wrong answer can score high, for example "within five business days" against a reference that says two differs by one word
  • a correct answer can score low if it is paraphrased or adds a helpful explanation
  • neither one checks whether the answer is supported by the retrieved passages

RAG triad

Three questions, where each one points to a different component to fix:

DimensionQuestion it answersComponent to fix
Context relevanceAre the retrieved passages relevant to the questionRetriever
FaithfulnessIs every claim in the answer supported by the passagesGenerator
Answer relevanceDoes the answer address the question that was askedPrompt or generator

Faithfulness

A claim is one atomic statement of fact. LLM-as-judge tools such as RAGAS split the answer into claims and check each one against the passages. Their verdicts should be calibrated against human labels.

Faithful is not the same as true

A true claim that the passages do not support still counts as unfaithful, because in an audit it cannot be defended.

Worked example

The passage says: "Cash deposits above 20,000 dollars require enhanced review by the compliance team within two business days." The answer says: "Deposits over 20,000 dollars trigger an enhanced review by compliance within two business days, and failures are reported to the police."

  • claim 1, deposits over 20,000 dollars trigger enhanced review: supported
  • claim 2, the review is done by compliance: supported
  • claim 3, it happens within two business days: supported
  • claim 4, failures are reported to the police: not in the passage

Faithfulness is . Claim 4 sounds plausible and might even be true, but nothing retrieved supports it, so it counts as a hallucination.

Code

A transparent stand-in for an LLM judge: a claim counts as supported if at least 60% of its content words appear in the passage.

import re
 
STOP = set("a an the is are be by of to in on for and or with within must that this it as at from".split())
 
def tokenize(text):
    return re.findall(r"[a-z0-9§\-]+", text.lower())
 
def content_words(text):
    """Tokens that carry meaning, with stop words removed."""
    return {t for t in tokenize(text) if t not in STOP}
 
def faithfulness(claims, passage, threshold=0.6):
    """Share of claims whose content words are mostly found in the passage."""
    passage_words = content_words(passage)
    results = []
    for claim in claims:
        words = content_words(claim)
        support = len(words & passage_words) / max(len(words), 1)
        results.append((claim, round(support, 2), support >= threshold))
    score = sum(r[2] for r in results) / len(results)
    return score, results
 
passage = "Clause CP-12 §4.3: cash deposits above 20,000 dollars require enhanced review by the compliance team within two business days."
claims = [
    "deposits over 20,000 dollars trigger enhanced review",
    "the review is done by compliance",
    "it happens within two business days",
    "failures are reported to the police",
]
score, results = faithfulness(claims, passage)
print(score)    # 0.75

Notes:

  • the answer must be split into atomic claims before scoring, which an LLM judge does automatically
  • word overlap cannot tell "within two business days" from "within five business days" if the words appear in the passage, and it misses paraphrased support, which is why LLM-as-judge methods exist

Faithfulness checks the answer, and the next question is whether to retrieve at all, see Long Context and Lost in the Middle.