RAG turns the idea "look up, then answer" into one probability: how likely is the answer , given the question . The retrieved passages act as hidden helpers, and each passage gets a vote that is weighted by how relevant it is.

There are 2 different encoders in this RAG system:

  1. question encoder
  2. passage encoder

Notation

SymbolMeaningExample
The user's question"What must happen before we onboard a senior government official?"
One passage from the document collectionPolicy passage on enhanced due diligence
The passages that the retriever returns for The 5 best-matching policy passages
The answer written by the LLM, as tokens"Senior management must approve the relationship"
Learnable parameters of the retrieverWeights of the question and passage encoders
Parameters of the generatorWeights of the language model

Read as "the probability of , given ".

Weighted vote over passages

Parts of the formula:

  • is the retriever: how relevant is passage to question , and these probabilities sum to 1 over the top-
  • is the generator: if the LLM reads passage , how likely is it to write answer
  • the sum lets every retrieved passage vote for the answer, weighted by its relevance
  • only the top- passages are summed, not the whole collection

A passage adds the most when it is both relevant (high ) and supports the answer (high ). It is like asking advisers and weighting each answer by how much you trust the adviser.

Where the retriever probabilities come from

The retriever encodes the question and each passage into vectors, scores them with a dot product, then a softmax turns the scores into probabilities:

Parts of the formula:

  • and are the question vector and the passage vector, see the dense retrieval step
  • is the dot product, a similarity score that is large when both vectors point the same way
  • the softmax makes every score positive and makes all scores sum to 1

For example, scores of become probabilities of about .

What the formula tells us

  • the retriever sets a ceiling: if the right passage is ranked 7th and , its term is missing from the sum, so the LLM never sees it however strong the LLM is
  • nobody labels which passage helped: the training data has only questions and answers, so the model learns from which passages make the correct answer more likely
  • parts are often off the shelf: in practice the retriever and the LLM are separate parts joined by a prompt, and the lesson still holds

If retrieval fails, nothing after it can recover

Measure the retriever on its own before blaming the LLM.

RAG-Sequence and RAG-Token

These are the 2 ways, proposed by Lewis et al. in 2020, to combine the passages when writing an answer:

PointRAG-SequenceRAG-Token
IdeaOne source per answerBlend the sources word by word
StepsPick a passage, write the whole answer from it, repeat for each passage, then average with the retrieval weightsFor every next word, consult all passages and blend their suggestions with the retrieval weights
AnalogyEach adviser writes a full answer and you weight the answers by trustThe advisers vote on every word of one shared answer
Best whenOne passage holds the whole answerThe answer needs facts from more than one passage

Here means the tokens already written, and multiplies the probabilities of the tokens.

Remembering tip

Look at where the sum sits. It sits outside the product in RAG-Sequence (choose the passage first), and inside the product in RAG-Token (mix the passages at every token).

Worked example

The question is "What is the cash threshold, and who reviews it?" and the answer has 2 tokens, = "20,000 dollars" and = "compliance team". Passage states the threshold and passage names the reviewer. The retriever gives and .

Probability of each answer token when the LLM reads each passage:

Passage
(threshold)0.90.2
(reviewer)0.40.9

Steps:

  • RAG-Sequence:
  • RAG-Token:

RAG-Sequence needs one passage to explain the whole answer, so it is penalised when each passage supports only part of it. RAG-Token takes from and from , so answers that combine several sources favour token-level mixing.

Code

import numpy as np
 
def softmax(scores):
    """Turn any list of scores into positive probabilities that sum to 1."""
    scores = np.asarray(scores, dtype=float)
    e = np.exp(scores - scores.max())   # subtracting the max avoids overflow, result is unchanged
    return e / e.sum()
 
print(np.round(softmax([2.0, 1.0, 0.5]), 2))   # [0.63 0.23 0.14]
 
p_z = np.array([0.7, 0.3])            # retriever: p(z1 | x), p(z2 | x)
 
# token_prob[j, i] = probability of answer token i when the LLM reads passage j
token_prob = np.array([[0.9, 0.2],
                       [0.4, 0.9]])
 
# RAG-Sequence: multiply over tokens for each passage, then weight by p(z) and sum
rag_sequence = np.sum(p_z * token_prob.prod(axis=1))
 
# RAG-Token: mix the passages for each token, then multiply over tokens
rag_token = np.prod(p_z @ token_prob)
 
print(f"RAG-Sequence = {rag_sequence:.4f}")   # 0.2340
print(f"RAG-Token    = {rag_token:.4f}")      # 0.3075

Notes:

  • prod(axis=1) multiplies across the tokens of each passage, which is the product inside the sum
  • p_z @ token_prob mixes the passages for every token, which is the sum inside the product
  • if both passages say the same thing (both rows equal), the 2 methods give the same value, so the gap only appears when the answer needs several passages

The retriever probability above comes from scoring passages, first by matching words, see Sparse Retrieval.