Long Context vs RAG: Testing Which Approach Finds the Right Evidence

The benchmark: same corpus, same questions, different evidence path

The practical contest between long context and retrieval-augmented generation is not about which architecture looks cleaner on a diagram. It is about whether a system can locate the decisive sentence, cite it faithfully and answer before users lose patience or budgets break. We built a controlled benchmark around 1,200 documents drawn from realistic enterprise collections: product manuals, contracts, policy updates, technical incident reports and quarterly filings. The corpus contained 8.7 million tokens, with individual task packs ranging from 80,000 to 900,000 tokens. Each of 360 questions required evidence from one to four passages, and 30 per cent included plausible but outdated or contradictory text.

Two systems received identical source material and prompts. The long-context system loaded an entire task pack into a model capable of accepting it in one request. The RAG system split documents into overlapping 600-token chunks, generated embeddings, retrieved 20 candidates and reranked them before sending the top eight to the same answer model. We measured evidence recall, citation precision, answer correctness, end-to-end latency and estimated API cost. Human reviewers marked the minimum supporting passages in advance, then verified whether each generated citation both existed and supported the claim attached to it.

This setup favoured neither approach completely. Long context avoided retrieval loss but had to reason over large, noisy inputs. RAG saw a smaller, more focused prompt but could never recover evidence that its retrieval stages missed. The benchmark therefore tested the entire evidence path rather than isolated model capability. That distinction matters: a model may understand a passage perfectly and still fail in production because the passage never reaches it.

Long context led on multi-document recall, but not uniformly

Across all questions, long context recovered 91 per cent of the gold evidence passages, compared with 84 per cent for RAG. Its advantage widened on synthesis tasks requiring three or more documents: 88 per cent recall versus 72 per cent. One procurement question, for example, required combining a renewal clause from a master agreement, a pricing exception in an annex and a termination date in later correspondence. The retrieval pipeline found the agreement and annex but ranked the email outside its final eight chunks. The long-context model saw all three and produced the correct timeline.

That result does not mean larger prompts automatically solve evidence discovery. Recall fell as irrelevant material accumulated. In packs below 150,000 tokens, long context reached 95 per cent evidence recall; above 600,000 tokens, it dropped to 86 per cent. Evidence positioned in the middle third of very large prompts was missed roughly twice as often as evidence near the beginning or end. The familiar ‘lost in the middle’ effect was reduced in newer models, not eliminated.

RAG performed best when questions contained distinctive entities, error codes or contractual phrases. For queries such as ‘What caused incident NET-1847?’ it achieved 96 per cent recall because the identifier created a strong retrieval signal. It struggled with indirect questions whose wording differed from the source. A policy document used ‘supplementary employment’, while the test question asked about ‘second jobs’; dense retrieval found semantically adjacent HR material but missed the governing paragraph. Hybrid keyword and vector search recovered four percentage points, showing that retrieval design, rather than the RAG label alone, determines much of the result.

Citation accuracy exposed a different winner

RAG produced more reliable citations even though it retrieved less evidence overall. Its citation precision was 94 per cent, against 87 per cent for long context. Because the answer model received eight numbered chunks, it could attach claims to a constrained set of sources. The long-context system navigated hundreds of pages and occasionally cited the correct document but the wrong section, or supplied a nearby passage that discussed the topic without proving the claim.

The distinction between a relevant citation and a supporting citation was critical. In a quarterly filing task, both systems correctly stated that operating margin had declined. The long-context answer cited the management commentary, which described cost pressure but contained no margin figure. RAG cited the table showing a fall from 18.4 to 16.9 per cent. A conventional relevance check would accept both; an evidence audit should accept only the latter. When reviewers demanded sentence-level support, long context’s apparent advantage in comprehensive reading did not translate into superior traceability.

Citation formatting also affected results. Requiring document IDs, page numbers and quoted evidence reduced unsupported claims by 28 per cent across both systems, although it added about 120 output tokens per answer. Long context benefited most from explicit evidence extraction before answer generation. A two-stage prompt that first created an evidence ledger raised citation precision from 87 to 92 per cent, but increased latency and cost. RAG achieved similar discipline through architecture: retrieval had already created a compact, addressable evidence set.

Answer quality depended on the shape of the task

Overall answer accuracy was close: 86 per cent for long context and 83 per cent for RAG. The three-point gap concealed sharper differences by task type. Long context led by 14 points on cross-document chronology, by 11 on policy conflict resolution and by nine on questions requiring the absence of a provision to be established across a document set. RAG led by seven points on pinpoint fact lookup and by five on repetitive technical corpora where many manuals shared similar structure.

The hardest cases involved versioning. A benefits question asked which parental-leave entitlement applied on 1 April 2025. The corpus contained policies from 2022, 2024 and 2025, plus an FAQ that had not been updated. Long context had access to every version and usually reconciled effective dates correctly. Baseline RAG often retrieved the older FAQ because its wording most closely matched the query. Adding metadata filters for jurisdiction and effective date lifted RAG accuracy on version-sensitive questions from 68 to 85 per cent.

Neither system was dependable when the task specification itself was vague. Questions asking for the ‘main risk’ encouraged synthesis beyond directly stated evidence, producing reviewer disagreement and unstable citations. Rewriting them as bounded requests—identify risks explicitly rated high, within the latest report—improved accuracy by 10 points for both approaches. Benchmark design therefore cannot be separated from product design. If users ask ambiguous questions, architecture alone will not create a single defensible truth.

Latency made full-corpus prompting difficult to hide

The median end-to-end latency for long context was 18.6 seconds, compared with 6.4 seconds for RAG when embeddings had been precomputed. At the 95th percentile, the figures were 41.2 and 12.7 seconds respectively. Input processing dominated the long-context path: sending 500,000 tokens before the model generated its first word imposed a delay that streaming could disguise only partially. RAG spent a median 430 milliseconds on retrieval and reranking, then passed approximately 7,000 tokens to the answer model.

Cold-start conditions narrowed the difference. When a new document set had to be parsed, chunked and embedded, RAG took 28 seconds before answering its first query. Long context required no index build and returned the first answer in 20 seconds. This makes full-context prompting attractive for ad hoc analysis of a newly uploaded deal room, investigation bundle or board pack, especially when users will ask only one or two questions.

Repeated use reversed the economics. Once a corpus received ten or more queries, indexing overhead was amortised and RAG’s lower per-query latency became decisive. Caching also helped both systems, but in different ways. Long context benefited when providers cached a stable prompt prefix; RAG benefited from cached retrieval results and reusable embeddings. Teams should test provider-specific cache rules carefully, because a small change to document ordering or system instructions can invalidate a long-context cache and restore the full latency penalty.

Cost favoured retrieval at scale, with important caveats

Using representative mid-2026 API pricing, the benchmark estimated a median long-context cost of £1.42 per query, versus £0.19 for RAG after indexing. At 100,000 monthly questions, that difference becomes roughly £123,000 before infrastructure, monitoring or human review. The precise numbers will move with model prices, but the underlying relationship is stable: repeatedly transmitting hundreds of thousands of tokens is expensive when only a few thousand are needed to answer.

RAG is not free infrastructure. Embedding 8.7 million tokens cost little in API terms, but production retrieval required a vector store, document parsers, access-control propagation, metadata maintenance, reranking and evaluation. For a small team analysing a fresh 300-page dossier once, engineering a durable pipeline would cost far more than the additional model tokens. For a support platform serving millions of searches against a stable knowledge base, the opposite is true.

Cost controls can also damage quality. Reducing RAG from eight final chunks to four cut answer-model input cost by 39 per cent but lowered evidence recall by six points on multi-source questions. Trimming long-context packs through crude document selection saved more than half the tokens yet recreated the central failure mode of retrieval: omitted evidence. The useful metric is not cost per query but cost per correct, cited answer. On that measure, RAG was 4.9 times more efficient overall, while long context remained competitive for low-volume, synthesis-heavy work.

The strongest production design was selective, not ideological

A hybrid router outperformed either fixed strategy. It used RAG by default, then escalated to long context when retrieval confidence was low, the question requested comparison across many documents or version metadata indicated potential conflict. This design achieved 89 per cent answer accuracy and 93 per cent citation precision, with a median cost of £0.34 per query. Only 17 per cent of questions triggered the expensive path, yet those escalations captured most of long context’s advantage on complex synthesis.

Confidence could not be reduced to the top retrieval score. Better signals included score separation between ranked chunks, diversity of source documents, coverage of named entities and whether the retrieved dates matched the question. If a query asked for changes between three policy editions but results came from one edition, the system escalated automatically. Conversely, a unique invoice number with a high lexical match stayed on the fast retrieval path. These rules were transparent enough for operations teams to inspect and tune.

The benchmark’s operational lesson is straightforward: test evidence handling, not model reputation. Build a question set from real user logs, label supporting passages, preserve document versions and report results by task type. Measure unsupported citations separately from incorrect answers, and publish latency at the 95th percentile rather than relying on averages. Long context is the stronger reader when the relevant evidence is dispersed and retrieval signals are weak. RAG is the stronger delivery mechanism when queries are frequent, evidence is identifiable and auditability matters. A routed system turns that trade-off into an engineering decision instead of a doctrinal argument.