The LLM Router Blueprint: Match Every Task to the Right Model

Routing Is an Operating System for Model Choice

An LLM router is the control layer that decides which model should handle each request. Instead of sending every prompt to the most capable and expensive frontier model, it classifies the task, applies policy, estimates difficulty and selects the cheapest option likely to meet the required standard. The destination might be a compact general model, a code specialist, a retrieval pipeline, a vision-language model or a frontier system with extended reasoning. The router can also reject the request, ask for clarification or escalate it for human review.

This matters because AI workloads are radically uneven. Extracting an invoice number from clean text may require fewer than 500 tokens and little reasoning; reviewing a 40-page contract for conflicting indemnity clauses requires long-context comprehension and careful judgement. Treating both requests alike wastes money at one end and creates unacceptable risk at the other. A well-designed router turns model diversity into an operational advantage, matching capability to demand rather than buying maximum intelligence for every transaction.

The strongest implementations separate routing from application logic. Product teams specify service requirements such as latency, cost, jurisdiction and minimum quality, while the routing layer maintains model capabilities, prices and observed performance. That separation allows a provider to be replaced or a new specialist to be introduced without rewriting the customer journey. It also prevents model names from becoming permanent business rules when prices and capabilities can change within weeks.

Start With Policy Rules, Not Model Benchmarks

Routing begins with hard constraints. Before predicting which model will produce the best answer, the system must determine which models are permitted to see the request. Data residency, retention terms, contractual restrictions, safety requirements and tool permissions can eliminate candidates immediately. A European healthcare workflow, for example, might allow patient data only through models hosted in an approved region with zero-retention terms. A coding assistant may use an external model for public repositories but require an internally hosted model for proprietary source code.

Rules should be explicit, versioned and auditable. A practical policy object might include tenant, data classification, task type, maximum latency, spend ceiling, required modalities and eligible providers. It can also specify whether the model may browse, execute code or call internal tools. Deterministic checks are preferable to asking another LLM whether a prompt contains sensitive information; classifiers can support detection, but the final policy decision should be reproducible.

Hard rules must remain distinct from preferences. “Never send regulated data outside the approved boundary” is a constraint; “prefer the fastest model” is an optimisation objective. Mixing the two encourages dangerous exceptions when a cheaper or higher-scoring model becomes available. The router should first produce an eligible set, then rank candidates within it. If that set is empty, it must fail safely rather than silently relaxing policy.

Classify Tasks Before Estimating Difficulty

The same model rarely leads every task category. A compact model may excel at intent classification, while a specialist outperforms it on SQL generation and a frontier model handles ambiguous research synthesis. The router therefore needs a task taxonomy grounded in production traffic: extraction, classification, summarisation, translation, code generation, mathematical reasoning, creative drafting, multimodal analysis and agentic tool use are useful starting points. Teams should add domain labels such as legal, medical or financial where terminology and risk justify separate evaluation.

Classification can combine deterministic signals with a lightweight model. File type, input length, requested output schema and the presence of images or source code are inexpensive clues. A small classifier can then interpret intent and estimate ambiguity. Consider two prompts containing the word “summarise”: one asks for five bullets from a 300-word press release; the other asks for a board-ready synthesis across 18 earnings transcripts. Their labels may match, but their context length, evidence burden and consequence of omission do not.

Difficulty estimation should use measurable features rather than prompt length alone. Indicators include the number of documents, required reasoning steps, domain specificity, tool dependencies, conflicting evidence and constraints on output. Historical outcomes add stronger evidence: if a compact model passes 98 per cent of extraction cases with fewer than ten fields but falls to 82 per cent above 30 fields, the router has a defensible escalation boundary. The taxonomy should evolve from observed failure modes, not from an abstract catalogue designed before traffic exists.

Use Confidence Signals Without Trusting Self-Confidence

A router needs confidence estimates at two stages: before execution, to choose a model, and after execution, to decide whether the result is acceptable. Pre-execution confidence can come from a trained routing classifier, nearest-neighbour comparisons with evaluated prompts or predicted success rates by task segment. If a compact model has a 96 per cent probability of meeting the rubric and the threshold is 95 per cent, it receives the request; if predicted success is 78 per cent, the router escalates.

Model self-ratings are weak evidence on their own. LLMs can sound certain while inventing facts, and verbal confidence often varies with prompting. Better post-execution signals include schema validation, citation coverage, retrieval entailment, compiler results, unit tests and agreement between independently generated answers. For customer-support classification, the output can be checked against an allowed label set. For code, the router can run tests. For a researched answer, it can verify that each factual claim is supported by retrieved passages.

Escalation should be designed as a normal path, not an exception. A fast model might draft an answer in 400 milliseconds; a verifier then scores factual support and instruction adherence. If the score falls below 0.9, a frontier model regenerates the response with the original context and the verifier’s critique. This cascade preserves savings on routine traffic while concentrating expensive inference on uncertain cases. The caveat is latency: a failed first attempt can make the final response slower than routing directly to the stronger model, so high-risk or clearly difficult requests should bypass the cascade.

Balance Cost, Latency and Quality With Explicit Thresholds

Routing is a constrained optimisation problem, not a contest for the highest benchmark score. The objective might be to minimise expected cost while maintaining at least 95 per cent task success and a p95 latency below three seconds. Expected cost must include input and output tokens, retrieval, tool calls, verification, retries and failed generations. A model priced at one-fifth of a frontier alternative can become more expensive if it triggers frequent escalations or produces verbose outputs.

A simple example shows the economics. Suppose 100,000 monthly requests cost 2 pence each on a frontier model, for a £2,000 inference bill. A router sends 70 per cent to a compact model at 0.3 pence, 20 per cent to the frontier model and 10 per cent through a two-stage path averaging 2.3 pence. The resulting cost is roughly £840, a 58 per cent reduction. Those savings are real only if quality remains within tolerance and additional infrastructure, evaluation and operational labour do not erase them.

Quality thresholds should vary by consequence. A marketing headline may tolerate subjective variation and route to a low-cost model unless the user requests strategic analysis. A payment dispute summary may require 99 per cent factual consistency, citation to source records and mandatory escalation when evidence conflicts. Organisations should define service tiers such as routine, important and critical, with distinct quality floors, latency budgets and human-review rules. One universal threshold is easy to administer but usually overpays for harmless tasks and underprotects consequential ones.

Build the Router as a Measurable Decision Pipeline

A production router typically follows a sequence: inspect and redact input, apply policy, classify the task, estimate complexity, filter eligible models, score candidates, execute, validate and escalate if required. Each decision should emit structured telemetry, including policy version, route selected, predicted success, model version, token usage, latency, validation outcome and final disposition. Without this trace, teams cannot determine whether a failure came from classification, model selection, retrieval or validation.

Candidate scoring can begin with a transparent weighted formula rather than a sophisticated learned system. For each eligible model, combine estimated quality, normalised cost, latency and reliability according to the service tier. A critical workflow might weight quality at 70 per cent, reliability at 20 per cent and cost at 10 per cent; a bulk categorisation job might reverse those priorities. As traffic grows, contextual bandits or supervised routers can learn better choices, but they require safeguards against exploration on high-risk requests.

Resilience is part of routing. Providers experience rate limits, regional outages and model regressions. The catalogue should record health, capacity and fallback compatibility, while circuit breakers stop sending traffic to a degraded endpoint. Fallbacks must be evaluated for the same task and policy constraints; replacing an unavailable European endpoint with a healthy service in another jurisdiction is not resilience but a compliance breach. Cache exact or semantic matches where appropriate, yet avoid serving stale answers for time-sensitive data such as stock availability or policy changes.

Evaluate the System, Not Just Individual Models

Offline evaluation should use a representative, stratified set of real tasks with expert-defined rubrics. Measure each candidate model by segment, then simulate routing policies to estimate quality, cost and latency. Aggregate accuracy can conceal dangerous weaknesses: a system scoring 94 per cent overall may achieve 99 per cent on simple extraction and only 71 per cent on rare but critical exception cases. Report pass rates by task, risk tier, language, document length and customer cohort.

Online evaluation closes the gap between laboratory tests and production behaviour. Run shadow routing to compare alternatives without exposing their outputs, followed by small canary releases. Monitor escalation rate, first-pass acceptance, user corrections, abandonment, tool-call failures and cost per successful task. A declining escalation rate is not automatically positive; the router may simply have become overconfident. Likewise, user thumbs-up data are useful but biased towards conspicuously good or bad experiences.

Regression controls must account for silent provider updates and prompt changes. Pin model versions where possible, maintain golden test sets and re-run evaluations whenever the routing policy, system prompt, retrieval stack or model catalogue changes. Define rollback triggers in advance, such as a two-point drop in groundedness or a 20 per cent rise in p95 latency. The unit of release is the whole decision pipeline, because a stronger model can still reduce system quality if the router sends it the wrong context or validation suppresses a correct answer.

Govern the Router as a Business-Critical Control Layer

Ownership should be shared but unambiguous. Platform teams can operate the router and model catalogue; domain teams should define rubrics and risk tiers; security and legal teams set non-negotiable policies; finance establishes budget limits. A routing council may sound bureaucratic, but uncontrolled local choices create duplicated integrations, inconsistent retention settings and no credible view of enterprise AI spend. The goal is not central approval for every experiment, but common guardrails and evidence standards.

Auditability becomes essential when a customer disputes an answer or a regulator asks how automated decisions were produced. Logs should show which policy applied, why a model was eligible, what validation ran and whether a human intervened, while respecting data-minimisation requirements. Teams also need clear retention periods and access controls for prompts and outputs. Storing every interaction indefinitely may simplify debugging, but it increases privacy exposure and can violate contractual commitments.

The mature blueprint is deliberately adaptable. Model prices fall, specialist systems improve and frontier capabilities spread into smaller models. Routing rules should therefore express desired outcomes rather than loyalty to a vendor: “use the lowest-cost eligible model with at least 97 per cent predicted success” is more durable than “send summaries to Model X”. Organisations that maintain robust evaluations, explicit thresholds and disciplined telemetry can change suppliers quickly while preserving quality. The competitive asset is not access to a particular model; it is the system that knows when each model is good enough.