RAG retrieves chunks, not whole documents. A good chunk can answer a question on its own.

Introduction

Example: a 400 page manual is about 200,000 tokens. With chunks of 512 tokens that overlap by 64 tokens, the manual gives 447 chunks.

Size of a chunk

  • too small: a chunk loses the context needed to understand it, for example "the threshold above" does not say which threshold
  • too large: one vector must summarise several topics, so the similarity is blurred, and each chunk uses more of the LLM context
  • overlap: repeating the end of one chunk at the start of the next keeps a sentence from being cut off from its context

Better ways to chunk

  • structure-aware chunking: split at headings, clauses, list items and tables, and prefix each chunk with its section title
  • parent-document retrieval: search with small chunks for precision, then hand the LLM the larger section around the match

Merging a family of near-identical chunks hurts exact questions

If 8 error code entries are merged into one long chunk, a question about one code matches the whole blob weakly, and the exact entry is no longer a separate unit that can be found.

Code

def chunk(tokens, size=512, overlap=64):
    """Slide a window over the token list. Each chunk starts size - overlap tokens after the last one."""
    step = size - overlap
    return [tokens[i:i + size] for i in range(0, max(len(tokens) - overlap, 1), step)]
 
tokens = list(range(200_000))      # stand-in for a 200,000 token manual
print(len(chunk(tokens)))          # 447

Notes:

  • range(0, len - overlap, step) stops before the last start would give a chunk made only of overlap
  • for text, split into tokens first (or words), then join each chunk back into a string before embedding

A single retriever is not enough on its own, so the next step is to combine two, see Reciprocal Rank Fusion (RRF).