Retrieval Augmented Generation (RAG) joins 2 parts: a retriever that finds useful text, and an LLM that writes the answer from that text. The model can "look up before it speaks".
Why retrieval is necessary
An LLM keeps its knowledge in its parameters. This knowledge has 2 problems:
- it is static, because the parameters stay frozen after training
- the model can hallucinate facts or miss recent information
RAG adds an external corpus, such as documents, databases or web pages. This corpus is called non-parametric memory, because the knowledge sits outside the model weights. You can update the corpus at any time and you do not retrain the model.
Remembering tip
LLM = writer, retriever = research assistant.
The assistant brings the sources first. The writer then writes the answer from these sources.
Structure
RAG has 2 steps:
- retriever: fetch the top-k documents that are most relevant to the user query
- generator: the LLM reads the query and the retrieved text together, then writes a grounded response
A short way to remember the order is "search first, then answer". The retrieved text adds to the internal knowledge of the model.
Data connection pipeline
The documents must be prepared and stored before the retriever can search them:
graph TD A[Source] --> B[Load] B --> C[Transform] C --> D[Embed] D --> E[Store] E --> F[Retrieve]
| Stage | What happens |
|---|---|
| Source | The raw data, such as PDFs, web pages, code or databases |
| Load | Read the raw data into documents |
| Transform | Split each document into smaller chunks |
| Embed | Turn each chunk into a vector |
| Store | Keep the vectors in a vector store |
| Retrieve | Embed the query and find the chunks with the closest vectors |
Frameworks
These are the 3 frameworks that we can use to build a RAG system:
- LangChain
- LlamaIndex
- Haystack
Example
An enterprise QA bot searches the internal documents of a company, then it answers questions about company policy.
RAG also lets a smaller LLM perform like a larger LLM, because much of the knowledge comes from the external corpus.
Benefits
Benefits of RAG:
- factual accuracy: the answer is grounded in real documents
- up-to-date answers: new documents go into the store with no retraining
- explainable sources: the answer can point to the documents that it used
Limitations
Limitations of RAG:
- retrieval latency: the search adds time before generation starts
- context length limits: only a fixed amount of retrieved text fits into the prompt
- irrelevant documents: a bad retrieval gives the generator the wrong context
- indexing and chunking: the answer quality depends on how the documents are indexed and split into chunks
RAG does not change the model weights. It changes only the text that the model reads at inference time.
RAG gives the model facts, and the next step is to make the model reason step by step over these facts, see LLM Reasoning Strategies.
To see how the retriever and the generator are combined in a formula, see RAG-Sequence and RAG-Token.