Dense retrieval judges relevance by matching meaning. Each text becomes an embedding, a vector of numbers (for example 384) trained so that texts with similar meanings get nearby vectors. It is the idea of Word Embeddings, now used for whole passages.
Structure
- retrieval becomes geometry: find the passage vectors that are closest to the question vector
- a bi-encoder uses 2 transformer encoders, one for questions and one for passages, so and
- relevance is the dot product
- passages are encoded once, offline, so a query needs only 1 encoder pass plus a nearest-neighbour search
For Question A, "senior government official" and "politically exposed person" end up close together, so the PEP policy is found.
Cosine and dot product
- cosine similarity measures the angle between 2 vectors, where 1 means the same direction and 0 means unrelated
- the dot product also grows with the length of the vectors
- after scaling every vector to length 1 (L2 normalisation), the dot product and the cosine are identical
- for unit vectors the Euclidean distance gives the same ranking, since
Practical rules:
- search with the same similarity that the model was trained with
- never mix vectors from different embedding models
- changing the embedding model means re-embedding the whole collection
Comparison with sparse retrieval
| Point | Sparse retrieval | Dense retrieval |
|---|---|---|
| Matches | Words | Meaning |
| Vector | Word counts, mostly zero | Full vector of about 384 numbers |
| Question A (paraphrase) | Misses the PEP policy | Finds it |
| Question B (exact code) | Finds the exact entry | Confuses KYC-E052 with KYC-E054, which read almost the same |
Remembering tip
Sparse reads the words, dense reads the meaning. Each one fails where the other succeeds, so real systems use both.
Code
These are the 2 ways to make the passage vectors:
LSA embeddings (offline, no download): TF-IDF followed by truncated SVD, then scaled to length 1. LSA is a simple stand-in for a neural bi-encoder. It puts words that co-occur in nearby directions, which helps paraphrases, and it compresses away rare tokens, which hurts exact codes.
import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.decomposition import TruncatedSVD
from sklearn.preprocessing import normalize
tfidf = TfidfVectorizer(sublinear_tf=True, stop_words="english")
X = tfidf.fit_transform(DOCS)
svd = TruncatedSVD(n_components=16, random_state=0) # compress to 16 dimensions
D = normalize(svd.fit_transform(X)) # offline: one unit vector per passage
embed_query = lambda q: normalize(svd.transform(tfidf.transform([q])))[0]Sentence-BERT (a neural bi-encoder, downloads about 90 MB, runs on CPU):
from sentence_transformers import SentenceTransformer
sbert = SentenceTransformer("all-MiniLM-L6-v2")
D = sbert.encode(DOCS, normalize_embeddings=True) # offline: one vector per passage
embed_query = lambda q: sbert.encode([q], normalize_embeddings=True)[0]Either way the search is the same:
def dense_search(query, k=50):
"""Return the indices of the k passages whose vectors are closest to the query vector."""
q = embed_query(query) # online: embed the question
return list(np.argsort(-(D @ q))[:k]) # dot product of unit vectors equals cosine similarityNotes:
normalize_embeddings=TruemakesD @ qa cosine similarity- the passage matrix
Dis built once, and only the question is embedded at query time - with LSA, the number of dimensions matters a lot for exact codes, since fewer dimensions throw away more rare tokens
A dense retriever is trained by contrast, see Contrastive Learning and InfoNCE Loss.