Re-ranking is a second, more careful look at the passages that the first retriever returned. A cross-encoder reads the question and one passage together, and gives a much better relevance score than the bi-encoder used for retrieval.

Comparison of the 2 encoders

PointBi-encoder (retrieval)Cross-encoder (re-ranking)
How it readsQuestion and passage separatelyQuestion and passage together, every word attending to every word
AnalogyJudging a book by its catalogue cardReading the book with the question in mind
Pre-compute passagesYes, once, offlineNo, one model pass per (question, passage) pair
Cost for 10M passagesOne query encoding plus a vector search10 million model passes, which is infeasible
AccuracyGoodMuch better

Two-stage pattern

  • retrieve the top 50 to 100 passages cheaply
  • re-rank only those passages with the cross-encoder
  • keep the few best passages for the LLM

Web search and recommender systems use the same funnel: cheap and wide first, then expensive and narrow.

Remembering tip

The bi-encoder is the catalogue card and the cross-encoder is reading the book. Use the card to pick 50 books, then read only those 50.

Why it matters

  • the right passage being somewhere in the top- does not mean the answer is right
  • a generator that leans on the top-ranked passage can answer from the wrong one, for example a definition of PEPs when the question asks what must happen before onboarding one
  • re-ranking puts the passage that really answers the question at the top

Code

from sentence_transformers import CrossEncoder
 
cross = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
def rerank(query, candidate_ids, top_n=5):
    """Score each (query, passage) pair together, then keep the best top_n."""
    pairs = [(query, DOCS[i]) for i in candidate_ids]
    scores = cross.predict(pairs)                      # one model pass per pair
    best = sorted(zip(candidate_ids, scores), key=lambda t: -t[1])
    return [i for i, _ in best[:top_n]]
 
top = rerank(query, hybrid_search(query, k=10))        # retrieve 10 cheaply, re-rank those 10

Notes:

  • the cost is one forward pass per candidate, so keep the candidate list short
  • the cross-encoder cannot be pre-computed, because its score depends on the question

Better candidates also come from a better question, see Query Transformation.