Retrieval metrics check the retriever on its own. If the final answer of a RAG system is wrong, the first question is whether the right passage was even retrieved.

Evaluate each stage separately

  • build a golden set: test questions, the IDs of the passages that answer them, and reference answers, written or checked by domain experts
  • retriever metrics: did the right passages reach the top-
  • generator metrics: given those passages, is the answer faithful to them and relevant to the question
  • end-to-end metrics: did the user get a correct answer
  • stage metrics tell you which box to fix, and end-to-end metrics tell you whether to ship

The 4 metrics

MetricThe question it answers
Recall@kDid the relevant passages make it into the top at all, which matters most for RAG
Precision@kWhat fraction of the top is relevant
MRR (mean reciprocal rank)How near the top is the first relevant passage, where 1 means always first, 0.5 means typically second, and 0 means never found
nDCGAre the most relevant passages at the very top, and it handles graded relevance such as grades 0 to 3

Here is the relevance grade at position , and IDCG is the DCG of the ideal order.

Worked example

Three queries, where the first relevant passage is at rank 1, rank 3, and not in the top 5.

  • MRR
  • Recall@5 is , and for the 3 queries
  • MRR and Recall are both pulled down by the third query, a complete miss, so the metrics point straight at the question to debug

For one query whose top 3 results have grades :

  • DCG
  • the ideal order gives IDCG
  • nDCG@3 , which says the ranking is nearly ideal, since the grade 2 passage belongs second and not third

Code

import numpy as np
import pandas as pd
 
def recall_at_k(ranked_ids, relevant, k):
    """Fraction of the relevant chunks that appear in the top k."""
    return len(relevant & set(ranked_ids[:k])) / len(relevant)
 
def reciprocal_rank(ranked_ids, relevant):
    """1 / rank of the first relevant chunk, or 0 if none is retrieved."""
    for rank, doc_id in enumerate(ranked_ids, start=1):
        if doc_id in relevant:
            return 1.0 / rank
    return 0.0
 
def ndcg_at_k(grades, k):
    """grades = relevance of each result, in ranked order."""
    dcg  = sum(g / np.log2(i + 2) for i, g in enumerate(grades[:k]))
    idcg = sum(g / np.log2(i + 2) for i, g in enumerate(sorted(grades, reverse=True)[:k]))
    return dcg / idcg if idcg > 0 else 0.0
 
print(round(np.mean([1/1, 1/3, 0]), 3))    # 0.444
print(round(ndcg_at_k([3, 0, 2], 3), 3))   # 0.939

Run every golden question through a retriever and average the results:

def evaluate(search_fn, name, k=3):
    rows = []
    for question, relevant, qtype in GOLD:             # (question, set of relevant IDs, type)
        ranked = [IDS[i] for i in search_fn(question)]
        first = next((r for r, d in enumerate(ranked, 1) if d in relevant), None)
        rows.append({"type": qtype,
                     f"recall@{k}": recall_at_k(ranked, relevant, k),
                     "RR": reciprocal_rank(ranked, relevant),
                     "first relevant rank": first})
    df = pd.DataFrame(rows)
    return df, {"retriever": name, f"Recall@{k}": df[f"recall@{k}"].mean(), "MRR": df["RR"].mean()}
 
bm25_df, bm25_sum = evaluate(sparse_search, "BM25")
dense_df, dense_sum = evaluate(dense_search, "Dense")
print(bm25_df.groupby("type")["RR"].mean())     # MRR per question type

Notes:

  • i + 2 appears in the nDCG code because enumerate starts at 0 while the formula uses for positions starting at 1
  • grouping by question type shows the pattern from the lecture: BM25 wins on exact IDs and the dense retriever wins on paraphrases
  • the GOLD list tags each question as exact-ID or paraphrase

Retrieval metrics do not judge the answer, which needs different metrics, see Faithfulness and the RAG Triad.