A long-context model can take a whole manual in its prompt, so is retrieval still needed. The answer depends on cost, volume and how well the model uses a long input.
Long context against RAG
Compare 2 ways to answer a question about a 200,000 token manual, at an example price of 3 dollars per million input tokens:
- long context: paste the whole manual, so each query uses 200,000 tokens and costs dollars
- RAG: send 5 chunks of 500 tokens plus 500 tokens of prompt and question, so each query uses 3,000 tokens and costs dollars
At 100,000 queries a day that is 60,000 dollars against 900 dollars, before counting the extra latency and the lack of per-user access control.
| Choose | When |
|---|---|
| Long context | One-off analysis of a single document |
| RAG | High-volume, multi-user systems |
Lost in the middle
Language models pay the most attention to the beginning and the end of their input, like a reader skimming a long report.
- Liu et al. placed the one relevant document at different positions among many documents
- accuracy was highest at the start or the end and lowest in the middle, a U-shaped curve
- so more retrieved chunks is not always better, since extra chunks add noise and push the evidence into the middle
- a bigger context window does not make retrieval quality irrelevant
Mitigations:
- re-rank and keep fewer, better chunks, see Re-ranking
- put the strongest evidence first
- remove near-duplicate chunks
Code
def query_cost(tokens_per_query, queries_per_day, price_per_million=3.0):
"""Daily cost in dollars for a given prompt size."""
return tokens_per_query * queries_per_day * price_per_million / 1_000_000
print(query_cost(200_000, 100_000)) # 60000.0 (whole manual in the prompt)
print(query_cost(5 * 500 + 500, 100_000)) # 900.0 (5 chunks of 500 tokens plus 500 for prompt and question)Retrieval can also be made smarter than a single shot, see Advanced RAG Patterns.