AI Observability Beyond Logs: Tracing Decisions, Costs, and Quality

Logs are evidence, not an explanation

A conventional application log can tell an operator that an endpoint returned a 500 error after 842 milliseconds. In a production AI system, the same record is radically incomplete. The request may have been rewritten, routed to one of three models, augmented with retrieved documents, passed through two safety checks and allowed to call a pricing tool before producing an answer. Even a technically successful 200 response can be operationally wrong: the output may cite stale policy, expose private context or cost 20 times more than expected. Observability therefore has to reconstruct the decision path, not merely record the final event.

The useful unit of analysis is an end-to-end trace containing linked spans for prompt assembly, retrieval, inference, tool use, validation and delivery. Each span should carry identifiers, timing, status and carefully selected metadata: model version, prompt template version, token counts, retrieval filters, tool schema and policy outcome. That structure lets an engineer distinguish a slow answer caused by vector search from one delayed by model generation. It also lets a product manager compare the quality and cost of two routing policies without stitching together records from six systems.

This does not mean capturing everything. Raw prompts may contain customer names, medical details or commercial secrets, while complete model outputs can create expensive, high-risk data stores. Mature teams separate telemetry from content: traces retain hashes, versions, categorical labels and aggregate measurements by default, with sampled, redacted payloads held under stricter access controls. The objective is sufficient evidence to explain behaviour without building a shadow archive of every conversation.

Trace the prompt that actually reached the model

Prompt monitoring must begin after assembly, not with the template stored in source control. Production prompts commonly combine system instructions, tenant rules, conversation history, retrieved passages and tool descriptions. A template labelled v17 may produce materially different inputs depending on truncation, localisation or the order in which context is inserted. Capture the template version, component hashes, total input tokens, per-component token allocation and any transformation applied. A sudden rise from 3,000 to 12,000 input tokens is often more actionable than the final latency figure.

Tracing also exposes prompt conflicts. Consider a support assistant instructed to issue refunds only after account verification. If a retrieved article says agents may waive verification for orders below £20, the model is receiving competing authority. The trace should show which instruction appeared where, which source supplied it and whether a policy layer overrode the answer. Teams can then test prompt hierarchy rather than vaguely blaming ‘hallucination’. Recording the final prompt fingerprint enables exact cohort comparisons while avoiding unnecessary retention of sensitive text.

Prompt telemetry needs release discipline. Treat prompts, model parameters and routing rules as a deployable configuration, each with immutable versions and rollback paths. Dashboards should segment outcomes by that configuration rather than averaging across all traffic. If v18 improves task completion from 78% to 84% but doubles refusal rates for non-native English speakers, the aggregate gain conceals a product regression. Observability is strongest when every behavioural change can be tied to a controlled release.

Make retrieval visible as a ranked decision

Retrieval-augmented generation is often monitored as a binary event: search succeeded or failed. That misses the failures that matter. A retriever can return five documents in 90 milliseconds and still supply the wrong evidence. Each retrieval span should record the query before and after rewriting, index and embedding versions, filters, candidate count, top-k results, relevance scores, reranking scores and document freshness. For access-controlled systems, it should also record whether permissions were applied before retrieval, not merely before display.

Quality requires more than similarity scores. Teams should measure context precision, context recall and citation faithfulness against labelled evaluation sets. Suppose a financial assistant answers 92% of benchmark questions correctly, yet only 68% of its cited passages support the claims made. That gap signals that the model is leaning on prior knowledge or inventing attribution. In production, proxy signals such as citation clicks, answer reformulations and zero-result rates can identify weak areas, but they should not be mistaken for ground truth.

Retrieval observability also clarifies trade-offs. Increasing top-k from five to 20 may lift recall by six percentage points while adding 6,000 tokens, 400 milliseconds and irrelevant context that lowers answer accuracy. A better intervention might be metadata filtering or reranking. By linking retrieval metrics to downstream quality, cost and latency within one trace, teams can determine whether extra context is genuinely useful rather than assuming that more evidence produces better answers.

Treat tool calls as distributed transactions

Once an agent can search databases, send emails or create refunds, observability becomes a control system. Every tool call needs a span containing the selected tool, arguments after validation, authorisation result, execution time, retries, returned data shape and side-effect status. The trace should distinguish between the model proposing an action and the platform executing it. That boundary matters when investigating whether an incorrect transfer originated in reasoning, schema validation, permissions or the downstream service.

Tool orchestration introduces failure modes hidden by overall success rates. An agent may call inventory three times because it cannot interpret a null response, or repeatedly attempt a payment after timeouts. Set budgets for calls, elapsed time and side effects; require idempotency keys for consequential operations; and expose loop detection as a first-class event. If the median task uses two calls but the 99th percentile uses 37, averages will conceal both cost leakage and customer risk.

Human approvals must be traced too. Record when an action entered review, the policy that triggered escalation, the reviewer’s decision and the delay introduced, without placing sensitive reasoning in unrestricted telemetry. This allows teams to measure whether controls are proportionate. A fraud workflow that blocks 0.2% of legitimate transactions may be acceptable; one that routes 18% of routine cases to a queue with a four-hour wait is an operational failure, even if the agent itself remains accurate.

Measure latency as a user journey

A single end-to-end latency number cannot explain an AI experience. Instrument queue time, prompt construction, retrieval, time to first token, generation time, tool execution and post-processing separately. Streaming systems need both time to first token and time to final answer: a response that begins in 700 milliseconds but takes 28 seconds to finish feels different from one that remains silent for eight seconds and completes at nine. Both may report similar averages while requiring different fixes.

Percentiles matter more than means. A median latency of 2.4 seconds can coexist with a 95th percentile of 14 seconds caused by overloaded model regions or slow tools. Break these distributions down by model, geography, tenant, task and input length. Service-level objectives should reflect product intent: perhaps 95% of simple classification requests must finish within 800 milliseconds, while a research assistant may have a 20-second budget if it provides progress and citations.

Tracing supports intelligent degradation. When retrieval exceeds its 500-millisecond budget, the system might use cached evidence; when a premium model is saturated, it might route low-risk tasks to a smaller model. These fallbacks must be labelled in traces and evaluated separately. A faster response is not an improvement if the fallback cuts answer accuracy from 90% to 72%. Latency policy should optimise the user outcome, not merely the stopwatch.

Attribute spending to decisions, not invoices

Provider invoices reveal total spend after the fact; observability should show why the money was spent. Calculate cost per trace from input tokens, output tokens, cached tokens, embeddings, reranking, tool infrastructure and retries. Then attribute it to a feature, tenant, workflow and outcome. A £30,000 monthly bill is difficult to act on. Knowing that abandoned document-analysis sessions consume £6.40 each because 200 pages are repeatedly embedded points directly to caching and session design.

Monitor distributions and unit economics rather than only daily totals. Cost per successful task is more useful than cost per request because cheap failures are not efficient. A smaller model costing £0.01 per attempt may need three retries and human review, while a £0.07 model completes the job first time. If the fully loaded figures are £0.23 and £0.09 respectively, routing everything to the cheapest model is false economy. Link spend to quality, escalation and conversion metrics.

Budgets should operate at several levels: per request, per user session, per tenant and per workflow. Alerts can detect a tenfold token increase after a prompt release, while hard limits prevent runaway agents from executing hundreds of calls. However, aggressive caps can truncate legitimate complex tasks. The trace should therefore record which budget was reached, what fallback occurred and whether the user recovered. Cost controls are product decisions with measurable quality consequences.

Turn output quality into an operating metric

Quality cannot be reduced to thumbs-up counts. Production evaluation needs a portfolio of measures: task correctness, groundedness, policy compliance, relevance, completeness, style and business outcome. Some require deterministic checks, such as valid JSON or matching an order number. Others need labelled datasets, specialist reviewers or model-based judges. Each score should be attached to the trace and segmented by use case; an overall score of 86 hides whether legal summaries improved while customer-support answers deteriorated.

Model-based evaluation scales, but it is not an oracle. Judges can favour verbose answers, share biases with the model under test and drift when their own version changes. Calibrate them against expert-labelled samples, version the rubric and judge, and report agreement rates. If a judge matches clinicians on 93% of straightforward cases but only 61% of ambiguous ones, use it for broad monitoring and route the uncertain cohort to human review. Confidence must determine how metrics are used.

Online signals complete the picture. Repeated questions, immediate edits, escalations, citation opens and task abandonment can reveal failures missed by offline benchmarks. Build alerts around meaningful changes, such as a five-point fall in groundedness for one document collection, rather than noisy fluctuations in a universal score. When quality, cost and latency share a trace, teams can choose deliberately: accept an extra 300 milliseconds for a seven-point accuracy gain, or reject it when conversion remains unchanged.

Build governance into the telemetry layer

AI traces are powerful and therefore sensitive. Apply data minimisation before ingestion: redact known personal fields, tokenise identifiers, encrypt payloads and separate operational metadata from sampled content. Access should be role-based and audited, with shorter retention for raw prompts than for aggregate metrics. A practical policy might keep full, redacted samples for seven days, structured trace metadata for 90 days and long-term aggregates without user content. Regulatory and contractual requirements may demand stricter boundaries.

Operational ownership is equally important. Platform teams should define trace schemas and instrumentation libraries, product teams should own task-level quality metrics, and security teams should set sampling and access policy. Establish a regular review that examines regressions across reliability, spend, safety and user outcome. Incidents should be reproducible from trace identifiers, and evaluation failures should feed release gates. If a model change raises unsafe tool proposals above an agreed threshold, deployment should stop automatically.

The strongest observability programme creates a closed loop: detect a change, isolate the responsible prompt, retrieval index, model or tool, reproduce it on a test set, ship a controlled correction and verify the result in production. Logs remain part of that system, but they are no longer the centre. The centre is a traceable account of what the AI saw, what it decided, what it cost and whether the result was good enough.