Retrieval Augmented Generation (RAG) joins 2 parts: a retriever that finds useful text, and an LLM that writes the answer from that text. The model can "look up before it speaks".

Why retrieval is necessary

An LLM keeps its knowledge in its parameters. This knowledge has 2 problems:

  • it is static, because the parameters stay frozen after training
  • the model can hallucinate facts or miss recent information

RAG adds an external corpus, such as documents, databases or web pages. This corpus is called non-parametric memory, because the knowledge sits outside the model weights. You can update the corpus at any time and you do not retrain the model.

Remembering tip

LLM = writer, retriever = research assistant.
The assistant brings the sources first. The writer then writes the answer from these sources.

Structure

RAG has 2 steps:

  • retriever: fetch the top-k documents that are most relevant to the user query
  • generator: the LLM reads the query and the retrieved text together, then writes a grounded response

A short way to remember the order is "search first, then answer". The retrieved text adds to the internal knowledge of the model.

Data connection pipeline

The documents must be prepared and stored before the retriever can search them:

graph TD
    A[Source] --> B[Load]
    B --> C[Transform]
    C --> D[Embed]
    D --> E[Store]
    E --> F[Retrieve]
StageWhat happens
SourceThe raw data, such as PDFs, web pages, code or databases
LoadRead the raw data into documents
TransformSplit each document into smaller chunks
EmbedTurn each chunk into a vector
StoreKeep the vectors in a vector store
RetrieveEmbed the query and find the chunks with the closest vectors

Frameworks

These are the 3 frameworks that we can use to build a RAG system:

  • LangChain
  • LlamaIndex
  • Haystack

Example

An enterprise QA bot searches the internal documents of a company, then it answers questions about company policy.

RAG also lets a smaller LLM perform like a larger LLM, because much of the knowledge comes from the external corpus.

Benefits

Benefits of RAG:

  • factual accuracy: the answer is grounded in real documents
  • up-to-date answers: new documents go into the store with no retraining
  • explainable sources: the answer can point to the documents that it used

Limitations

Limitations of RAG:

  • retrieval latency: the search adds time before generation starts
  • context length limits: only a fixed amount of retrieved text fits into the prompt
  • irrelevant documents: a bad retrieval gives the generator the wrong context
  • indexing and chunking: the answer quality depends on how the documents are indexed and split into chunks

RAG does not change the model weights. It changes only the text that the model reads at inference time.

RAG gives the model facts, and the next step is to make the model reason step by step over these facts, see LLM Reasoning Strategies.

To see how the retriever and the generator are combined in a formula, see RAG-Sequence and RAG-Token.