Beyond Hallucinations: The AI Reliability Risks Teams Underestimate

Hallucinations are only the visible failure mode

Generative AI’s reputation for inventing facts has produced a useful but incomplete model of risk. A fabricated court case or imaginary citation is conspicuous, memorable and often easy to classify as failure. More dangerous errors frequently look ordinary: an accurate answer based on last year’s policy, a workflow that drops one field in 200, or a summary that omits the only sentence requiring urgent action. These outputs are fluent, plausible and structurally compatible with the systems receiving them. That makes them less likely to trigger scrutiny and more likely to travel.

Reliability should therefore be measured against the job, not merely against factual truth. A customer-support assistant can quote every product specification correctly and still fail if it routes a cancellation request to sales. A coding agent can produce valid code that quietly bypasses an audit log. A document extractor can achieve 98 per cent field accuracy yet create unacceptable exposure if the missing 2 per cent clusters around liability caps or renewal dates. The central question is not whether a model sometimes makes things up. It is whether the combined model, data, tools and human process fail safely when uncertainty, novelty or ambiguity appears.

Stale knowledge can be more persuasive than falsehood

Many AI systems answer from a mixture of training data, retrieved documents and instructions supplied at runtime. Each layer has an expiry problem. A model may know an old tax threshold; a retrieval system may rank a superseded policy above its replacement; an internal knowledge base may preserve a former employee’s workaround long after the process changed. The resulting answer can be perfectly grounded in an available source and still be operationally wrong. Because it includes dates, citations and company terminology, users may trust it more than an unsupported response.

Consider a procurement assistant reviewing supplier terms after a regulatory update. If the company’s repository contains 40 policy files and only 37 carry reliable effective dates, semantic search may retrieve the closest wording rather than the current rule. The system can then recommend a clause that legal approved 18 months ago but has since prohibited. The remedy is not simply a larger model. Teams need document owners, effective and expiry dates, version lineage, deletion rules and retrieval tests built around real questions. A citation proves provenance, not currency; showing the source date beside the answer is often as important as showing the source itself.

Freshness also involves a trade-off. Continuously indexing every new file increases coverage but can admit drafts, duplicates and malicious content. Restricting retrieval to curated material improves authority but slows updates. Mature systems make this policy explicit: which sources are authoritative, how quickly changes must propagate, and what the assistant should say when no current source exists. “I cannot verify the present rule” is a reliable outcome when the alternative is polished obsolescence.

Tool use turns language errors into actions

An isolated chatbot produces text. An agent connected to email, databases, payment systems or deployment pipelines changes the world. Reliability then depends on more than the model’s reasoning: API schemas, permissions, network responses, retries, timeouts and state all become part of the decision. A tool may return an empty list because no records exist, because authentication expired or because a query parameter was malformed. If the agent interprets all three as “nothing to do”, failure remains silent.

Retries create another class of risk. If a payment request times out after the transaction succeeds, an agent may submit it again unless the operation has an idempotency key. If a CRM update partially succeeds, the model may continue with an inaccurate view of customer state. Even tiny error rates compound across chains. A workflow with eight independent steps, each 99 per cent reliable, has only about a 92 per cent chance of completing without a step failure. Independence is an optimistic assumption; outages and schema changes often break several steps together.

Safe tool use requires constrained interfaces rather than broad authority. Separate read from write permissions, validate arguments outside the model, cap transaction values, require confirmation for irreversible actions and record both intended and observed outcomes. High-impact operations should use a two-phase pattern: prepare a proposed action, then approve and execute it. The additional click or review queue reduces speed, but that friction is valuable where a plausible sentence can otherwise become a cancelled order, deleted account or public message.

Formatting failures break otherwise correct automation

Teams often dismiss formatting as cosmetic until a downstream parser rejects an answer or, worse, accepts it incorrectly. Models can add commentary around JSON, change an enum from “high” to “urgent”, represent a date as 04/05/26, or omit a required key. To a person, the intent remains obvious. To software, the output may be invalid, ambiguous or mapped to the wrong action. Reliability at the interface boundary is therefore a product requirement, not a prompt-writing detail.

Structured-output modes and schema validation reduce this risk, but they do not establish semantic correctness. A response can satisfy a schema while placing a gross price in the net-price field or assigning 0 to a missing quantity. Systems should distinguish absent, unknown and genuinely zero values; validate ranges and cross-field relationships; and reject impossible combinations. If an invoice total is £12,000 while extracted line items sum to £1,200, the workflow needs an exception rather than a syntactically immaculate record.

The strongest designs make invalid states difficult to express. Use typed function calls, enumerated values, locale-specific date rules and deterministic transformations for calculations. Keep the model responsible for interpretation, not arithmetic or serialisation that conventional code handles better. Teams should also test formatting under long inputs, unusual characters and partial context, because production failures tend to appear at boundaries that tidy demonstrations never reach.

Silent omissions evade ordinary accuracy metrics

A summary can contain no false statements and still be dangerously incomplete. Models optimise for relevance and compression, which means they routinely discard details that appear secondary. In a clinical handover, that detail might be an allergy. In a contract review, it might be an automatic renewal clause. In a security report, it might be the one host for which scanning failed. Conventional fact-checking asks whether each included claim is supported; it does not ask whether every required claim was included.

This distinction matters when teams advertise an extraction system as 97 per cent accurate. If a document contains 100 fields and three are missed, the headline sounds strong. If one missed field is a termination deadline worth £500,000, average accuracy is irrelevant. Evaluation should weight omissions by consequence and measure recall separately for critical categories. A useful test set includes sparse documents, contradictory clauses, scanned pages, tables split across pages and examples where the correct output is “not present”.

Workflows can also expose omissions structurally. Require the system to account for every page or section, display unfilled mandatory fields, and produce coverage indicators tied to source spans. For high-risk review, use checklists derived from policy rather than asking for a generic summary. A second model may help challenge the first, but correlated blind spots limit its value. Independent rules, source-level reconciliation and targeted human review are usually stronger than simply generating another fluent opinion.

Automation bias makes acceptable systems unsafe

Reliability is partly a human-factors problem. Once an AI recommendation appears inside an established interface, users tend to treat it as system output rather than a fallible suggestion. Confidence rises when the text is fast, detailed and professionally phrased, none of which reliably signals correctness. Reviewers also become less vigilant after dozens of good results. A nominal “human in the loop” can therefore become a rubber stamp, especially when performance targets reward throughput rather than detected errors.

Imagine an operations team processing 300 AI-drafted decisions per shift. If the interface asks staff to approve each one and fewer than 2 per cent usually need changes, attention will decline. Adding a reviewer does not create meaningful control when the reviewer has 20 seconds, cannot see the source evidence or is penalised for escalation. Better design directs scarce attention towards uncertainty and impact: sample routine cases, route high-value or anomalous cases for deeper review, and require users to record a reason when overriding or approving sensitive recommendations.

Displaying a model-generated confidence score is not enough; such scores are often poorly calibrated and may encourage false precision. More useful signals include missing sources, conflicting evidence, unusual input length, tool failures and distance from known examples. Organisations should also measure reviewer behaviour: approval rates, time spent, disagreement patterns and whether errors are caught before or after execution. Human oversight must be tested as a control, not asserted as a feature.

Reliability engineering needs consequence-based tests

A serious evaluation programme begins with a failure inventory. Teams should map where knowledge can be stale, tools can misfire, schemas can drift, information can disappear and people can over-trust the result. They should then rank scenarios by severity, frequency and detectability. A low-frequency error that transfers money without an alert deserves more investment than a common wording defect users immediately notice. Aggregate benchmark scores obscure this distinction.

Production testing should combine fixed regression suites with continuous sampling. Golden examples catch repeat failures; adversarial cases probe ambiguity, prompt injection and malformed files; shadow runs compare proposed actions with actual human decisions before automation is enabled. Metrics should include critical-field recall, tool-call success, invalid-output rate, abstention quality, source freshness and time to detection. Every model, prompt, retrieval-index or API change needs a versioned release and a rollback path. Otherwise teams cannot explain why behaviour shifted on Tuesday.

The commercial trade-off is straightforward: controls add latency, cost and occasional refusals. Yet unrestricted automation externalises those costs into corrections, customer harm and incident response. A sensible architecture allocates assurance according to consequence. Low-risk drafting can tolerate lightweight checks; financial, legal, medical and security actions require stronger validation, narrower permissions and explicit escalation. The most reliable AI system is not the one that always produces an answer. It is the one that knows when evidence, interfaces or oversight are insufficient to act.