Re-ranking is a second, more careful look at the passages that the first retriever returned. A cross-encoder reads the question and one passage together, and gives a much better relevance score than the bi-encoder used for retrieval.
Comparison of the 2 encoders
| Point | Bi-encoder (retrieval) | Cross-encoder (re-ranking) |
|---|---|---|
| How it reads | Question and passage separately | Question and passage together, every word attending to every word |
| Analogy | Judging a book by its catalogue card | Reading the book with the question in mind |
| Pre-compute passages | Yes, once, offline | No, one model pass per (question, passage) pair |
| Cost for 10M passages | One query encoding plus a vector search | 10 million model passes, which is infeasible |
| Accuracy | Good | Much better |
Two-stage pattern
- retrieve the top 50 to 100 passages cheaply
- re-rank only those passages with the cross-encoder
- keep the few best passages for the LLM
Web search and recommender systems use the same funnel: cheap and wide first, then expensive and narrow.
Remembering tip
The bi-encoder is the catalogue card and the cross-encoder is reading the book. Use the card to pick 50 books, then read only those 50.
Why it matters
- the right passage being somewhere in the top- does not mean the answer is right
- a generator that leans on the top-ranked passage can answer from the wrong one, for example a definition of PEPs when the question asks what must happen before onboarding one
- re-ranking puts the passage that really answers the question at the top
Code
from sentence_transformers import CrossEncoder
cross = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
def rerank(query, candidate_ids, top_n=5):
"""Score each (query, passage) pair together, then keep the best top_n."""
pairs = [(query, DOCS[i]) for i in candidate_ids]
scores = cross.predict(pairs) # one model pass per pair
best = sorted(zip(candidate_ids, scores), key=lambda t: -t[1])
return [i for i, _ in best[:top_n]]
top = rerank(query, hybrid_search(query, k=10)) # retrieve 10 cheaply, re-rank those 10Notes:
- the cost is one forward pass per candidate, so keep the candidate list short
- the cross-encoder cannot be pre-computed, because its score depends on the question
Better candidates also come from a better question, see Query Transformation.