A long-context model can take a whole manual in its prompt, so is retrieval still needed. The answer depends on cost, volume and how well the model uses a long input.

Long context against RAG

Compare 2 ways to answer a question about a 200,000 token manual, at an example price of 3 dollars per million input tokens:

  • long context: paste the whole manual, so each query uses 200,000 tokens and costs dollars
  • RAG: send 5 chunks of 500 tokens plus 500 tokens of prompt and question, so each query uses 3,000 tokens and costs dollars

At 100,000 queries a day that is 60,000 dollars against 900 dollars, before counting the extra latency and the lack of per-user access control.

ChooseWhen
Long contextOne-off analysis of a single document
RAGHigh-volume, multi-user systems

Lost in the middle

Language models pay the most attention to the beginning and the end of their input, like a reader skimming a long report.

  • Liu et al. placed the one relevant document at different positions among many documents
  • accuracy was highest at the start or the end and lowest in the middle, a U-shaped curve
  • so more retrieved chunks is not always better, since extra chunks add noise and push the evidence into the middle
  • a bigger context window does not make retrieval quality irrelevant

Mitigations:

  • re-rank and keep fewer, better chunks, see Re-ranking
  • put the strongest evidence first
  • remove near-duplicate chunks

Code

def query_cost(tokens_per_query, queries_per_day, price_per_million=3.0):
    """Daily cost in dollars for a given prompt size."""
    return tokens_per_query * queries_per_day * price_per_million / 1_000_000
 
print(query_cost(200_000, 100_000))          # 60000.0 (whole manual in the prompt)
print(query_cost(5 * 500 + 500, 100_000))    # 900.0   (5 chunks of 500 tokens plus 500 for prompt and question)

Retrieval can also be made smarter than a single shot, see Advanced RAG Patterns.