RAG retrieves chunks, not whole documents. A good chunk can answer a question on its own.
Introduction
Example: a 400 page manual is about 200,000 tokens. With chunks of 512 tokens that overlap by 64 tokens, the manual gives 447 chunks.
Size of a chunk
- too small: a chunk loses the context needed to understand it, for example "the threshold above" does not say which threshold
- too large: one vector must summarise several topics, so the similarity is blurred, and each chunk uses more of the LLM context
- overlap: repeating the end of one chunk at the start of the next keeps a sentence from being cut off from its context
Better ways to chunk
- structure-aware chunking: split at headings, clauses, list items and tables, and prefix each chunk with its section title
- parent-document retrieval: search with small chunks for precision, then hand the LLM the larger section around the match
Merging a family of near-identical chunks hurts exact questions
If 8 error code entries are merged into one long chunk, a question about one code matches the whole blob weakly, and the exact entry is no longer a separate unit that can be found.
Code
def chunk(tokens, size=512, overlap=64):
"""Slide a window over the token list. Each chunk starts size - overlap tokens after the last one."""
step = size - overlap
return [tokens[i:i + size] for i in range(0, max(len(tokens) - overlap, 1), step)]
tokens = list(range(200_000)) # stand-in for a 200,000 token manual
print(len(chunk(tokens))) # 447Notes:
range(0, len - overlap, step)stops before the last start would give a chunk made only of overlap- for text, split into tokens first (or words), then join each chunk back into a string before embedding
A single retriever is not enough on its own, so the next step is to combine two, see Reciprocal Rank Fusion (RRF).