From Prompt to Production: A Reliable LLM Application Architecture

Start with a system, not a prompt

A dependable LLM application is not a clever prompt wrapped in an interface. It is a distributed software system in which a probabilistic model performs selected tasks under explicit constraints. The production architecture must decide what context the model receives, which model handles each request, what tools it may call, how its output is validated and what happens when any component fails. Prompts matter, but they sit alongside conventional controls such as authentication, rate limits, queues, schemas, audit logs and service-level objectives.

Consider a customer-support assistant serving 500,000 conversations a month. A demonstration might send every message to the largest available model and display the answer. A production service first identifies language, intent, account permissions and risk. A password-reset query may follow a deterministic workflow; a billing dispute may retrieve policy and account data; a threat of self-harm or legal action may be escalated immediately. Treating all three as free-form generation wastes money and introduces avoidable risk.

Define success in operational terms before selecting models. Targets might include a 95th-percentile latency below four seconds, grounded-answer accuracy above 92 per cent, schema compliance above 99.5 per cent and a human-escalation rate below 8 per cent. These measures turn architecture debates into engineering decisions. A more capable model is useful only if its gains justify additional latency, cost and operational complexity.

Route requests by complexity and risk

Routing is the control plane of an LLM application. It directs requests to a model, workflow or human according to task complexity, confidence, sensitivity and service tier. A small classifier can distinguish summarisation, retrieval, extraction, transactional actions and unsafe content. Rules should handle high-certainty cases, while ambiguous requests move to a stronger model. This arrangement prevents a premium reasoning model from spending expensive tokens on greetings, simple categorisation or template completion.

A practical three-tier design might use a sub-second model for intent classification and extraction, a mid-range model for most grounded responses and a high-capability model for multi-document analysis. If 70 per cent of traffic is resolved by the first two tiers, blended inference cost can fall by 40–70 per cent compared with sending everything to the top tier. The trade-off is routing error: an underpowered model may produce a plausible but incomplete answer. Teams should therefore test routers against labelled traffic and bias borderline cases towards escalation.

Risk must override efficiency. Requests involving financial transfers, medical advice, regulated disclosures or destructive tool calls require stricter policies than product-search questions. Route using both static attributes, such as user role and jurisdiction, and dynamic signals, such as model confidence, detected intent and retrieved evidence quality. Record every routing decision with the policy version so that an incident can be reconstructed rather than debated from incomplete logs.

Build retrieval around evidence quality

Retrieval-augmented generation can reduce hallucination, but only when the retrieval layer supplies relevant, current and authorised evidence. The pipeline begins before vector search: documents need ownership, effective dates, access controls, stable identifiers and a deletion process. Chunking should follow semantic structure rather than arbitrary character counts. A 700-token policy section with its heading and scope is usually more useful than several fragments that separate exceptions from the rule they qualify.

Hybrid retrieval combines keyword search, embeddings and metadata filters. Keyword search captures exact product codes and legal phrases; embeddings find conceptual matches; metadata limits results by market, customer plan or publication status. A reranker can then select the best five passages from 30 candidates. Evaluate retrieval independently using metrics such as recall at 10 and mean reciprocal rank. If the correct evidence is absent, changing the generation prompt will not repair the system.

The application should require citations that resolve to the source passages shown to the model. It should also distinguish no evidence from negative evidence. If a benefits database contains no answer about overseas dental cover, the system must not interpret silence as exclusion. Configure a minimum retrieval score, ask clarifying questions when filters are uncertain and return an explicit ‘insufficient evidence’ response when necessary. Refusal is a reliability feature when the alternative is confident invention.

Make structured outputs the contract

Free-form text is suitable for conversation, not for connecting models to software. When an LLM selects an action, extracts fields or produces data for another service, its output should conform to a versioned schema. A support workflow might require an intent enum, customer identifier, confidence score, evidence references and proposed next action. Schema-constrained decoding can prevent malformed JSON, but validation must still check business rules: a refund amount cannot be negative, and a closed account cannot receive a new subscription.

Separate generation from execution. The model may propose a tool call such as ‘issue_refund’ with an order ID and amount, but a deterministic service must verify permissions, refund limits, order status and idempotency before acting. For a £20 refund, the system may allow automatic execution; above £100, it might require human approval. This boundary limits the model’s authority and makes security controls testable using ordinary application methods.

Design for repair rather than assuming perfect compliance. If validation fails, return concise machine-readable errors to the model for one controlled retry. Repeated failures should trigger a fallback or queue the case for review, not create an unbounded loop. Keep raw output, parsed output, validation errors and final action as separate records. That distinction is invaluable when determining whether a defect came from the model, parser, policy engine or downstream API.

Observe every step, not just the answer

Traditional monitoring reports whether a service is up; LLM observability must explain whether it is useful, grounded and safe. Trace each request across routing, retrieval, prompt construction, model calls, tool use, validation and delivery. Capture latency, token counts, model and prompt versions, retrieval identifiers, cache status, retry count and outcome. Sensitive data should be redacted or tokenised before logging, with retention periods aligned to regulation and customer contracts.

Quality requires both offline and online evaluation. Maintain a representative test set covering common tasks, edge cases, adversarial instructions and policy-sensitive scenarios. Run it whenever a prompt, model, embedding index or retrieval configuration changes. In production, sample conversations for automated checks such as citation support and schema validity, then use trained reviewers for nuanced criteria. Averages are insufficient: segment results by language, customer tier, channel and task because a 94 per cent overall score can conceal a serious failure in a smaller market.

Operational alerts should connect metrics to action. A rise in retrieval misses may indicate an indexing failure; higher input tokens may reveal prompt growth; increased refusals may follow an overly restrictive safety update. Distributed traces shorten diagnosis by showing the exact passage, tool response and model version involved. Without this evidence, teams tend to patch prompts based on anecdotes, creating regressions that surface days later.

Use caching without serving stale truth

Caching is one of the most effective controls for both cost and latency. Exact-match caching works for repeated, stable requests such as policy summaries, while semantic caching can reuse answers for paraphrases. Prompt-prefix caching reduces the cost of large, repeated instructions, and retrieval caches avoid rerunning common searches. At scale, lifting the cache hit rate from 10 to 35 per cent can remove millions of model tokens and materially improve tail latency.

The difficult question is invalidation. Cache keys should include the model, prompt version, tenant, permissions, locale and relevant knowledge-base version. A cached answer about delivery times must expire when logistics policy changes; an account-specific answer must never cross customer boundaries. Time-to-live settings should reflect volatility: minutes for stock levels, hours for service status and perhaps days for approved product descriptions. Sensitive or action-triggering responses may be unsuitable for semantic caching altogether.

Measure cache quality, not merely hit rate. Track stale-answer incidents, permission mismatches and the proportion of reused responses that reviewers would still approve. Consider caching intermediate artefacts rather than final prose: retrieved document IDs, normalised entities or validated classifications can be safer to reuse. The best cache strategy reduces duplicate computation while preserving the principle that current evidence outranks a previously generated answer.

Engineer fallbacks and human review

Failure is normal: providers throttle requests, models time out, tools return partial data and safety filters reject legitimate language. A resilient design uses bounded retries with jitter, circuit breakers and provider-aware rate limits. Fallbacks should preserve task semantics, not simply swap model names. If the primary model supports a large context window or strict tool schemas, the secondary path must either provide equivalent capabilities or deliberately narrow the task.

Graceful degradation is often better than silent substitution. When account systems are unavailable, the assistant can provide general policy information and state that it cannot confirm the customer’s balance. When retrieval fails, it can offer contact options rather than generating from memory. For non-urgent workloads, queues absorb traffic spikes and allow later processing. For interactive requests, latency budgets should cap retries; one additional attempt may be reasonable, while three sequential attempts can turn a two-second response into an unacceptable delay.

Human review belongs inside the architecture, with clear triggers and feedback loops. Escalate when confidence is low, sources conflict, requested actions exceed thresholds or policy detects regulated risk. Reviewers need the user request, retrieved evidence, proposed answer, validation results and a concise reason for escalation. Their decisions should become labelled data for evaluation and routing improvements, not disappear into a support queue. The dependable application is therefore not fully autonomous: it automates routine work, exposes uncertainty and reserves consequential judgement for accountable people.