Retrieval metrics check the retriever on its own. If the final answer of a RAG system is wrong, the first question is whether the right passage was even retrieved.
Evaluate each stage separately
- build a golden set: test questions, the IDs of the passages that answer them, and reference answers, written or checked by domain experts
- retriever metrics: did the right passages reach the top-
- generator metrics: given those passages, is the answer faithful to them and relevant to the question
- end-to-end metrics: did the user get a correct answer
- stage metrics tell you which box to fix, and end-to-end metrics tell you whether to ship
The 4 metrics
| Metric | The question it answers |
|---|---|
| Recall@k | Did the relevant passages make it into the top at all, which matters most for RAG |
| Precision@k | What fraction of the top is relevant |
| MRR (mean reciprocal rank) | How near the top is the first relevant passage, where 1 means always first, 0.5 means typically second, and 0 means never found |
| nDCG | Are the most relevant passages at the very top, and it handles graded relevance such as grades 0 to 3 |
Here is the relevance grade at position , and IDCG is the DCG of the ideal order.
Worked example
Three queries, where the first relevant passage is at rank 1, rank 3, and not in the top 5.
- MRR
- Recall@5 is , and for the 3 queries
- MRR and Recall are both pulled down by the third query, a complete miss, so the metrics point straight at the question to debug
For one query whose top 3 results have grades :
- DCG
- the ideal order gives IDCG
- nDCG@3 , which says the ranking is nearly ideal, since the grade 2 passage belongs second and not third
Code
import numpy as np
import pandas as pd
def recall_at_k(ranked_ids, relevant, k):
"""Fraction of the relevant chunks that appear in the top k."""
return len(relevant & set(ranked_ids[:k])) / len(relevant)
def reciprocal_rank(ranked_ids, relevant):
"""1 / rank of the first relevant chunk, or 0 if none is retrieved."""
for rank, doc_id in enumerate(ranked_ids, start=1):
if doc_id in relevant:
return 1.0 / rank
return 0.0
def ndcg_at_k(grades, k):
"""grades = relevance of each result, in ranked order."""
dcg = sum(g / np.log2(i + 2) for i, g in enumerate(grades[:k]))
idcg = sum(g / np.log2(i + 2) for i, g in enumerate(sorted(grades, reverse=True)[:k]))
return dcg / idcg if idcg > 0 else 0.0
print(round(np.mean([1/1, 1/3, 0]), 3)) # 0.444
print(round(ndcg_at_k([3, 0, 2], 3), 3)) # 0.939Run every golden question through a retriever and average the results:
def evaluate(search_fn, name, k=3):
rows = []
for question, relevant, qtype in GOLD: # (question, set of relevant IDs, type)
ranked = [IDS[i] for i in search_fn(question)]
first = next((r for r, d in enumerate(ranked, 1) if d in relevant), None)
rows.append({"type": qtype,
f"recall@{k}": recall_at_k(ranked, relevant, k),
"RR": reciprocal_rank(ranked, relevant),
"first relevant rank": first})
df = pd.DataFrame(rows)
return df, {"retriever": name, f"Recall@{k}": df[f"recall@{k}"].mean(), "MRR": df["RR"].mean()}
bm25_df, bm25_sum = evaluate(sparse_search, "BM25")
dense_df, dense_sum = evaluate(dense_search, "Dense")
print(bm25_df.groupby("type")["RR"].mean()) # MRR per question typeNotes:
i + 2appears in the nDCG code becauseenumeratestarts at 0 while the formula uses for positions starting at 1- grouping by question type shows the pattern from the lecture: BM25 wins on exact IDs and the dense retriever wins on paraphrases
- the
GOLDlist tags each question as exact-ID or paraphrase
Retrieval metrics do not judge the answer, which needs different metrics, see Faithfulness and the RAG Triad.