The Hidden Cost of AI: A Field Guide to Tokens, Latency, and Scaling

The invoice starts with tokens, not intelligence

AI pricing looks deceptively simple: multiply input and output tokens by a published rate. Suppose a service handles 1 million requests a month, each carrying 2,000 input tokens and producing 500 output tokens. That is 2 billion input tokens and 500 million output tokens before retries, retrieval or internal reasoning. At illustrative rates of $3 per million input tokens and $15 per million output tokens, the monthly model bill is $13,500. The arithmetic is easy; establishing what actually enters the arithmetic is not.

Token economics reward disciplined product design. A verbose system prompt repeated on every request may cost more than the user’s text. Conversation history grows with each turn, while structured outputs, citations and explanations inflate generation. Output tokens are commonly priced above input tokens and take longer to produce, so a 1,500-token answer can be more damaging than a 1,500-token prompt. Teams should therefore measure tokens by feature, customer, model, route and outcome—not merely report one aggregate bill.

Caching changes the picture, but only when prompts contain stable prefixes and providers offer favourable cached-input rates. A 10,000-token policy manual shared across thousands of calls may become economical when cached; a personalised document rebuilt on every request will not. Compression, summarisation and prompt deduplication can cut spend, yet each introduces failure modes. The cheapest token is the one never sent, provided removing it does not reduce task success and trigger expensive corrections later.

Retries turn edge cases into budget lines

Production requests rarely travel directly from user to model and back. Timeouts, malformed JSON, safety refusals, tool failures and rate limits create retries. A system with a nominal 1 million monthly calls and a 7 per cent retry rate actually makes 1.07 million calls. If an orchestration layer retries twice, while the application and gateway each retry once, a single fault can fan out into several paid requests. Without idempotency controls, tool calls may also repeat real-world actions such as issuing refunds or creating tickets.

Retries are not equally costly. A connection failure before generation may incur little or no usage; a validation failure after a 2,000-token answer incurs almost the full cost, followed by another attempt. Asking the model to repair malformed output can be cheaper than regenerating from scratch, but it adds latency and another failure surface. Constrained decoding, schema validation and shorter maximum outputs often deliver better economics than indiscriminate retry logic.

Track first-attempt success, retry count and tokens consumed per successful user outcome. A route that costs two pence per call but succeeds only 70 per cent of the time can be dearer than a three-pence route that succeeds at 98 per cent. Set retry budgets by error class, add exponential back-off with jitter, and stop retrying deterministic failures such as an oversized prompt. Reliability engineering is cost engineering when every attempt is metered.

Long context is rented memory

Large context windows encourage teams to treat prompts as storage. Entire contracts, chat histories and codebases are appended because the model can accept them. Capacity, however, is not the same as efficiency. Sending 100,000 input tokens at $3 per million costs $0.30 per request before output. At 100,000 requests a month, that is $30,000 simply to reread information, much of which may be irrelevant.

Long contexts also raise latency and can reduce answer quality. Time to first token generally increases as the model processes more input, while important evidence may be diluted among duplicated or conflicting passages. A support assistant does not need the customer’s complete five-year history to answer a delivery query. It may need the latest order, current tracking event, refund policy and two recent messages. Selecting those items is both an economic and editorial decision about what evidence deserves attention.

Context management should be explicit. Keep recent turns verbatim, summarise older ones, retrieve durable facts from a store and preserve links to source records. Summaries save tokens but may erase qualifiers, so sensitive workflows should retain auditable originals and refresh summaries when facts change. Establish context budgets by task—for example, 8,000 tokens for routine support and 40,000 for contract review—rather than allowing every feature to consume the model’s maximum window.

Retrieval has its own cost stack

Retrieval-augmented generation is often presented as the inexpensive alternative to fine-tuning or giant prompts. It can be, but retrieval is not free. Documents must be parsed, cleaned, chunked, embedded, indexed and updated. Queries may be rewritten, embedded and sent to a vector database, then reranked before selected passages reach the model. Each stage adds infrastructure charges, engineering work and latency.

Consider a knowledge base containing 10 million chunks. Embedding it once may be modest compared with generation, but frequent document changes, duplicate ingestion and multiple embedding versions can multiply storage and compute. Query-time costs can include an embedding call, vector search, metadata filtering and a reranker. If retrieval adds 250 milliseconds and returns ten chunks of 500 tokens, it also adds 5,000 input tokens to the final prompt. Poor recall then causes the model to answer incorrectly or the user to ask again.

The trade-off is quality per retrieved token. Hybrid keyword and vector search often outperforms either alone for identifiers, product codes and exact policy language. Reranking can reduce ten candidates to three strong passages, lowering prompt cost enough to offset its own charge. Measure retrieval precision, answer grounding and citation validity, not just search latency. A cheap index that repeatedly supplies irrelevant text is an expensive component disguised as infrastructure.

Latency is a product cost, not a dashboard metric

Users experience latency as doubt. A search suggestion that takes four seconds feels broken; a detailed legal analysis delivered in 20 seconds may feel impressive. The acceptable budget depends on the interaction. Time to first token governs perceived responsiveness in chat, while total completion time matters for workflows that cannot proceed until valid structured output arrives. Streaming improves perception but does not reduce token charges and may complicate moderation or cancellation.

Latency accumulates across the chain: authentication, retrieval, prompt assembly, model queueing, generation, tool execution and post-processing. A route with 150 milliseconds of retrieval, 600 milliseconds to first token and six seconds of generation may become a 12-second experience after two sequential tool calls. Parallelising independent operations can help, whereas agentic loops—plan, call, observe, revise—can turn one user action into five model round trips.

Faster models often cost less, but not always, and they may require more retries on difficult tasks. Routing is therefore preferable to choosing one model for everything. Use a small model for classification, extraction and query rewriting; escalate ambiguous or high-value cases to a stronger model. Set service-level objectives at the feature level, cap generation lengths, and monitor p50, p95 and p99 latency. An acceptable median can conceal a tail that drives abandonment and support complaints.

Observability can rival the workload it watches

AI systems generate unusually rich telemetry: prompts, completions, token counts, retrieval traces, tool arguments, evaluations and user feedback. Recording everything appears prudent until storage, indexing and data-governance bills arrive. One million traces containing 20 KB each produce about 20 GB before replicas, indexes and retention overhead. Add full retrieved documents, multi-step agent spans and six months of retention, and the footprint can grow by an order of magnitude.

The greater risk is collecting sensitive content without a defined purpose. Prompts may contain personal data, health details, source code or commercial contracts. Redaction adds compute and can miss context-dependent secrets; encryption and access controls add operational burden. Sampling lowers cost, but naive sampling can discard rare failures. A sensible policy keeps complete traces for errors and high-risk routes, samples routine successes, and stores aggregated token and latency metrics for the full population.

Observability should answer operational questions: which tenant is driving spend, which prompt version increased retries, which retrieval source causes unsupported claims, and what percentage of costly requests produce a useful outcome? Trace identifiers must connect gateway, model, database and tool activity. Cost attribution should include model usage, search, storage and external APIs. Otherwise teams optimise the visible model line while hidden services quietly consume the margin.

Traffic shape determines the architecture

Monthly averages conceal the capacity problem. Ten million requests spread evenly across 30 days equal roughly four requests per second; the same volume concentrated into weekday business hours or a product launch may create peaks hundreds of times larger. Providers impose requests-per-minute and tokens-per-minute limits, while reserved capacity and higher service tiers carry commitments. A viral feature can therefore fail despite having a sustainable monthly budget.

Concurrency is driven by arrival rate and response time. If 50 requests arrive each second and remain active for ten seconds, the system must support roughly 500 concurrent requests. Long generations hold connections, memory and worker slots. Queues smooth bursts but introduce waiting; aggressive autoscaling protects latency but may raise infrastructure costs. Batch processing can dramatically improve throughput for offline classification or enrichment, yet it is unsuitable when users expect immediate answers.

Design for degradation rather than binary availability. During peaks, shorten context, disable optional reranking, route low-risk tasks to a smaller model, defer background summaries or place non-urgent work in a queue. Tenant quotas and spend limits prevent one customer from exhausting shared capacity. Load tests should reproduce token distributions, tool delays and retry behaviour, not merely send identical short prompts. Real traffic is heavy-tailed, and the most expensive one per cent of requests often dictates the limits.

Unit economics must follow successful outcomes

The durable metric is cost per successful outcome: resolved support case, approved document, completed booking or retained subscriber. Assume an assistant costs $0.04 in model usage, $0.01 in retrieval and infrastructure, and $0.01 in observability. With a 10 per cent retry rate, the direct expected cost rises towards $0.066 before engineering, support and payment fees. If only 75 per cent of sessions resolve the user’s need, cost per resolution is about $0.088—not the four cents advertised by the model tariff.

Revenue and risk determine whether that figure is attractive. An eight-pence assistant that deflects a £6 support contact has ample room for quality safeguards. The same assistant may be uneconomic for a free consumer feature with weak retention. Conversely, spending £1 on a careful model and verification pipeline can be rational when reviewing a £100,000 transaction. Cost controls must reflect the value and consequence of each decision.

Build a cost ledger that allocates tokens, retries, retrieval, tools, storage and human review to features and customers. Pair it with quality measures and run changes as experiments: does a shorter prompt save 18 per cent while holding resolution steady; does reranking add 80 milliseconds but cut repeat contacts by 6 per cent; does a premium model improve conversion enough to justify its rate? AI economics are managed through these measured trade-offs, not through token price alone.