RAG turns the idea "look up, then answer" into one probability: how likely is the answer , given the question . The retrieved passages act as hidden helpers, and each passage gets a vote that is weighted by how relevant it is.
There are 2 different encoders in this RAG system:
- question encoder
- passage encoder
Notation
| Symbol | Meaning | Example |
|---|---|---|
| The user's question | "What must happen before we onboard a senior government official?" | |
| One passage from the document collection | Policy passage on enhanced due diligence | |
| The passages that the retriever returns for | The 5 best-matching policy passages | |
| The answer written by the LLM, as tokens | "Senior management must approve the relationship" | |
| Learnable parameters of the retriever | Weights of the question and passage encoders | |
| Parameters of the generator | Weights of the language model |
Read as "the probability of , given ".
Weighted vote over passages
Parts of the formula:
- is the retriever: how relevant is passage to question , and these probabilities sum to 1 over the top-
- is the generator: if the LLM reads passage , how likely is it to write answer
- the sum lets every retrieved passage vote for the answer, weighted by its relevance
- only the top- passages are summed, not the whole collection
A passage adds the most when it is both relevant (high ) and supports the answer (high ). It is like asking advisers and weighting each answer by how much you trust the adviser.
Where the retriever probabilities come from
The retriever encodes the question and each passage into vectors, scores them with a dot product, then a softmax turns the scores into probabilities:
Parts of the formula:
- and are the question vector and the passage vector, see the dense retrieval step
- is the dot product, a similarity score that is large when both vectors point the same way
- the softmax makes every score positive and makes all scores sum to 1
For example, scores of become probabilities of about .
What the formula tells us
- the retriever sets a ceiling: if the right passage is ranked 7th and , its term is missing from the sum, so the LLM never sees it however strong the LLM is
- nobody labels which passage helped: the training data has only questions and answers, so the model learns from which passages make the correct answer more likely
- parts are often off the shelf: in practice the retriever and the LLM are separate parts joined by a prompt, and the lesson still holds
If retrieval fails, nothing after it can recover
Measure the retriever on its own before blaming the LLM.
RAG-Sequence and RAG-Token
These are the 2 ways, proposed by Lewis et al. in 2020, to combine the passages when writing an answer:
| Point | RAG-Sequence | RAG-Token |
|---|---|---|
| Idea | One source per answer | Blend the sources word by word |
| Steps | Pick a passage, write the whole answer from it, repeat for each passage, then average with the retrieval weights | For every next word, consult all passages and blend their suggestions with the retrieval weights |
| Analogy | Each adviser writes a full answer and you weight the answers by trust | The advisers vote on every word of one shared answer |
| Best when | One passage holds the whole answer | The answer needs facts from more than one passage |
Here means the tokens already written, and multiplies the probabilities of the tokens.
Remembering tip
Look at where the sum sits. It sits outside the product in RAG-Sequence (choose the passage first), and inside the product in RAG-Token (mix the passages at every token).
Worked example
The question is "What is the cash threshold, and who reviews it?" and the answer has 2 tokens, = "20,000 dollars" and = "compliance team". Passage states the threshold and passage names the reviewer. The retriever gives and .
Probability of each answer token when the LLM reads each passage:
| Passage | ||
|---|---|---|
| (threshold) | 0.9 | 0.2 |
| (reviewer) | 0.4 | 0.9 |
Steps:
- RAG-Sequence:
- RAG-Token:
RAG-Sequence needs one passage to explain the whole answer, so it is penalised when each passage supports only part of it. RAG-Token takes from and from , so answers that combine several sources favour token-level mixing.
Code
import numpy as np
def softmax(scores):
"""Turn any list of scores into positive probabilities that sum to 1."""
scores = np.asarray(scores, dtype=float)
e = np.exp(scores - scores.max()) # subtracting the max avoids overflow, result is unchanged
return e / e.sum()
print(np.round(softmax([2.0, 1.0, 0.5]), 2)) # [0.63 0.23 0.14]
p_z = np.array([0.7, 0.3]) # retriever: p(z1 | x), p(z2 | x)
# token_prob[j, i] = probability of answer token i when the LLM reads passage j
token_prob = np.array([[0.9, 0.2],
[0.4, 0.9]])
# RAG-Sequence: multiply over tokens for each passage, then weight by p(z) and sum
rag_sequence = np.sum(p_z * token_prob.prod(axis=1))
# RAG-Token: mix the passages for each token, then multiply over tokens
rag_token = np.prod(p_z @ token_prob)
print(f"RAG-Sequence = {rag_sequence:.4f}") # 0.2340
print(f"RAG-Token = {rag_token:.4f}") # 0.3075Notes:
prod(axis=1)multiplies across the tokens of each passage, which is the product inside the sump_z @ token_probmixes the passages for every token, which is the sum inside the product- if both passages say the same thing (both rows equal), the 2 methods give the same value, so the gap only appears when the answer needs several passages
The retriever probability above comes from scoring passages, first by matching words, see Sparse Retrieval.