Why AI Summaries Lose Crucial Details—and How to Test for It

A summary can be accurate and still be unsafe

Most summary failures are not obvious falsehoods. The system usually identifies the subject, preserves the broad argument and produces fluent prose. The danger lies in selective loss. A 40-page policy may become five convincing paragraphs that retain its purpose while omitting the clause that limits eligibility to contracts signed before 31 March. A meeting recap may capture the decision to launch but drop the condition that legal approval is required first. Readers see a coherent account and assume completeness where only plausibility has been delivered.

This distinction matters because many organisations evaluate summaries as if they were short answers. Reviewers ask whether the output is generally correct, readable and relevant. Those criteria reward thematic fidelity, not operational fidelity. If a summary says a supplier must report incidents, it sounds accurate; if the source says incidents must be reported within 24 hours, through a specified portal, with four mandatory data fields, the missing details determine whether the obligation can actually be met. The summary has not invented a claim, yet it may still cause a breach.

The risk rises as compression increases. Reducing 10,000 words to 1,000 leaves room for qualifications and examples; reducing the same source to 150 words forces aggressive selection. That trade-off is unavoidable, but unmanaged omission is not. The central evaluation question should be: which facts would change a reader’s decision, action or interpretation if they were absent?

Why models preserve themes but drop constraints

Large language models are optimised to predict useful continuations and are often tuned to favour concise, coherent answers. Broad themes are repeated across a document and therefore receive strong statistical support. Constraints may appear once, in a footnote, table, parenthesis or subordinate clause. The phrase “the warranty lasts three years” is easy to retain; “except for batteries, which are covered for 12 months” competes for limited space and may look secondary despite carrying substantial commercial significance.

Document structure compounds the problem. Important material is not always signposted as important. A regulatory consultation may place the headline proposal in the executive summary and the exceptions in an annex. A loan agreement may define “Business Day” near the front, then rely on that definition throughout. Tables can encode thresholds that disappear when converted poorly from PDF. OCR errors may turn “0.5%” into “5%”, while page headers interrupt sentences and make dependencies harder to recover. The model cannot preserve a detail it never receives cleanly.

Prompts can also encourage omission. Requests such as “give me the key points” or “summarise briefly” leave importance undefined. Models tend to select information that explains what the document is about, not everything a user must do. A better instruction names the classes of detail that deserve protection: deadlines, quantities, exceptions, responsibilities, approval gates, penalties, definitions and unresolved issues. Even then, prompting is a control, not proof of performance. Testing must establish what survives consistently.

The details most likely to disappear

Exceptions are especially fragile because they reverse or narrow a nearby rule. Consider: “Employees may work remotely up to three days a week, except during probation, formal performance management or the final two weeks of each quarter.” A weak summary preserves the three-day policy and loses all three exclusions. Negation produces a related hazard. “No approval is required for purchases below £500 unless the supplier is new” can be compressed into a dangerously simple claim that purchases below £500 need no approval.

Numbers create another failure class. Dates, percentages, ranges, currencies and units may be dropped, rounded or attached to the wrong subject. A report that says revenue rose 12%, while operating profit fell 4%, can become “financial performance improved”. A clinical protocol specifying 5 milligrams per kilogram may be rendered as 5 milligrams. The language remains polished while the operational meaning changes. Lists are vulnerable too: a model may preserve four of seven required documents without signalling that the list is incomplete.

Obligations and ownership are often softened. “The processor shall notify the controller within 12 hours” may become “incidents should be reported promptly”, erasing the responsible party, recipient, deadline and mandatory force. Modal verbs matter: must, may, should and will are not interchangeable. So do dependencies: a project is not approved merely because a budget exists if approval also depends on a security review. Evaluators should treat these elements as structured facts rather than stylistic details.

Build a test set around decision-critical claims

A useful evaluation begins with representative source documents, not generic benchmark passages. Select material from the workflows where summaries will be used: contracts, board papers, research reports, support cases, policy manuals or call transcripts. Include clean and messy inputs, short and long documents, tables, appendices and scanned pages. A pilot set of 50 documents can expose major patterns; a production programme may need 200 to 500 examples stratified by document type and risk.

For each document, human reviewers should create a “must-preserve” ledger. Every entry records a source span, a normalised claim and a criticality rating. One contract might yield: termination requires 30 days’ notice; notice must be written; early termination triggers a £25,000 fee; the fee does not apply after a material breach; notices must be sent to a named address. The ledger should distinguish explicit facts from reasonable interpretations, because penalising a model for failing to infer an unstated conclusion corrupts the test.

Criticality should reflect consequences. A three-tier scheme is practical: Tier 1 details could cause legal, financial, safety or compliance harm; Tier 2 details could materially affect a decision or workflow; Tier 3 details improve context but are not action-determining. Reviewers should also add “must-not-claim” items covering tempting but unsupported interpretations. This turns evaluation from a vague judgement of quality into a controlled comparison between source, expected content and output.

Score omission separately from distortion

A single quality score conceals the difference between missing information and incorrect information. At minimum, measure critical-detail recall: the proportion of must-preserve claims represented in the summary. If a document contains 20 required claims and the output retains 16, recall is 80%. Weighting by criticality gives a more realistic result. Missing a decorative example should not offset preserving a safety exception. One simple weighting assigns five points to Tier 1, three to Tier 2 and one to Tier 3.

Next, score claim fidelity. A retained detail can still be altered through wrong numbers, reversed negation, misplaced ownership or weakened modality. Use labels such as entailed, partially entailed, contradicted and unsupported. Then calculate hallucination rate separately: how many summary claims lack support in the source? A system with 95% critical-detail recall but frequent invented conclusions may be less trustworthy than one with 90% recall and near-zero unsupported claims.

Report results by detail type and document class, not only as an average. An overall recall of 88% may hide 97% performance on topics but 61% on exceptions and 54% on table-derived quantities. Set release thresholds that reflect use. For low-stakes article previews, 85% weighted recall may be acceptable. For compliance summaries, organisations may require 98% recall on Tier 1 claims, zero contradicted Tier 1 claims and mandatory human review whenever the system expresses uncertainty or the source extraction fails.

Use adversarial pairs to expose brittle behaviour

Standard documents reveal ordinary failure; adversarial pairs reveal whether the system understands distinctions it tends to blur. Create two nearly identical passages that differ in one consequential feature: “within 30 days” versus “after 30 days”; “may disclose” versus “must disclose”; “more than £10,000” versus “£10,000 or more”. Summarise both and check whether the outputs preserve the difference. If they converge on the same wording, the summariser is insensitive to the feature under test.

Counterfactual editing is particularly efficient because it controls for topic and style. Take a real policy and change one exception, threshold or responsible party while leaving everything else intact. Run multiple prompt versions and model settings. A robust system should update only the affected part of the summary. This method also catches memorised or template-driven language: a model may continue to say “managers approve expenses” even after the source assigns approval to finance.

Stress tests should vary the position and presentation of critical details. Put the same deadline in the opening paragraph, a footnote, a table and the final page. Express £1.5 million as “£1,500,000” and as “one and a half million pounds”. Add distractor figures nearby. Test a list with six obligations, then eight. Run each case several times if generation is non-deterministic. Variance is itself a risk signal: a detail preserved in seven of ten runs is not dependable enough for an automated compliance workflow.

Design the product around verifiable summaries

Evaluation should shape the interface, not remain a laboratory exercise. Summaries are safer when important claims link to source passages. Citations let users verify a deadline or exception without searching 80 pages, although citation presence alone does not prove that all critical material was included. A useful design pairs concise prose with a structured panel for dates, amounts, obligations, exceptions and open questions. This reduces pressure on one narrative paragraph to carry every operational detail.

Systems should disclose compression and coverage. Showing “180-word summary of a 12,400-word document” helps users calibrate expectations. For high-risk material, the product can flag that three Tier 1 clauses were detected and confirm whether each appears in the output. When extraction confidence is low because a table was unreadable or pages were missing, the system should refuse to present a seamless summary. Friction is preferable to false assurance.

Finally, monitor failures after deployment. Sample outputs by document type, investigate user corrections and rerun the benchmark whenever the model, prompt, parser or document pipeline changes. A parser update can damage table handling even if the language model remains unchanged. The strongest programme treats omission recall as an operational metric alongside latency and cost. A summary is not reliable because it sounds informed; it is reliable when testing shows that the details governing action survive compression.