Semantic caching is retrieval with consequences
A conventional cache answers a simple question: have we seen this exact key before? Semantic caching asks a harder one: have we seen a request that means roughly the same thing? Instead of hashing the raw prompt, the application embeds a normalised representation of the request, searches a vector index, and reuses a stored response when similarity exceeds a threshold. “How do I reset my password?” might therefore match “I cannot remember my login password”, even though their strings differ completely. The reward is lower model spend and faster responses; the risk is returning an answer that is linguistically related but operationally wrong.
That distinction makes semantic caching suitable for repetitive, bounded workloads rather than every model call. Customer-support explanations, product FAQs, document summaries and natural-language-to-SQL requests over stable schemas can benefit. Personalised financial guidance, rapidly changing inventory answers and tool calls with side effects deserve far more caution. A cache hit on a £0.02 generation completed in 40 milliseconds rather than 1.8 seconds looks attractive, but one incorrect answer can erase savings through support costs, lost trust or regulatory exposure. Treat the cache as a retrieval system whose output can affect users, not as an invisible performance switch.
Build a cache key that reflects the real task
Embedding the user’s latest sentence alone is usually a design error. The semantic key should represent every input that materially changes the correct answer: the user request, system policy, product or workspace, language, model family, tool availability, relevant conversation state and knowledge-base version. A support question about “export limits” may require different answers for free and enterprise plans. Either include the plan in the embedded text or, preferably, store it as metadata and require an exact metadata match before vector similarity is considered.
Normalise only what is genuinely irrelevant. Lowercasing, trimming repeated whitespace and replacing volatile request IDs can improve recall. Removing numbers, dates, negation or entity names can be disastrous. “Cancel order 431” is not interchangeable with “cancel order 918”, and “does this contain peanuts?” must never match “does this contain no peanuts?”. A practical record might contain an embedding of the canonical request, the generated answer, tenant ID, locale, entitlement tier, policy version, source-document hashes, creation time, expiry time and model identifier. Store the original request as well; without it, engineers cannot audit surprising matches.
Embedding choice also matters. Use the same embedding model for writes and reads, and version it explicitly. A compact model may cost fractions of a penny per thousand queries and add 10–30 milliseconds, but its similarity geometry still needs testing on your domain. Switching embedding models without rebuilding the index makes historical scores incomparable. For multilingual traffic, verify cross-language behaviour rather than assuming it: a model that links English and French questions well may still confuse product names or technical terminology.
Set thresholds from labelled pairs, not instinct
Cosine similarity is not a universal confidence score. A threshold of 0.90 can be overly strict in one embedding space and dangerously permissive in another. Build an evaluation set containing three classes: requests that should share an answer, requests that are related but require different answers, and clearly unrelated requests. Include hard negatives such as “upgrade my plan” versus “cancel my plan”, “refund pending” versus “refund denied”, and questions that differ only by region, date or account state. Several hundred labelled pairs can reveal more than weeks of adjusting a threshold in production.
For each candidate threshold, calculate cache-hit rate, precision among accepted hits and the financial impact of errors. Suppose 10,000 daily requests produce 3,000 candidates above 0.86, but manual evaluation finds 150 wrong reuses. Raising the threshold to 0.92 might reduce accepted hits to 2,000 while cutting wrong answers to 20. The second configuration saves less compute, yet its 99 per cent hit precision may be commercially superior to 95 per cent. In high-risk domains, 99 per cent may still be inadequate.
A single global threshold is rarely optimal. Use stricter thresholds for billing, identity, legal or safety questions and looser ones for low-stakes explanations. Add a grey zone: accept matches above 0.94, generate a fresh answer below 0.88, and rerank or validate candidates between those values. Comparing entities, intent, structured parameters and retrieved source IDs can catch errors that vector distance misses. The best policy combines semantic similarity with deterministic constraints.
Isolate tenants, users and conversational state
Cross-user leakage is the most serious implementation failure. A semantically similar request must not retrieve an answer containing another customer’s name, balance, case history or internal documents. Enforce tenant isolation in the vector query itself through separate indexes, namespaces or mandatory metadata filters. Do not retrieve globally and filter afterwards; an indexing bug, log entry or fallback path can still expose data. For highly sensitive workloads, maintain per-user partitions or disable response reuse entirely.
Shared caching remains possible when responses are demonstrably generic. Divide the system into a public layer for approved, non-personalised material and a private layer for tenant- or user-specific outputs. A question such as “What file formats can I upload?” can use a shared entry if the policy is universal. “Why did my upload fail?” probably depends on file metadata, account limits and scan results, so its key and namespace must reflect that context. Redact secrets before embedding because vector stores, telemetry systems and backups can all become disclosure surfaces.
Conversation history requires equal discipline. “What about the annual option?” means nothing without the preceding discussion. Summarise the relevant state into a stable task representation or cache only self-contained requests. Include authorisation and entitlement versions as exact filters, not merely semantic text. If a user loses access to a document, old cached answers derived from it must become unreachable immediately, regardless of their time-to-live.
Design invalidation before pursuing hit rate
Semantic cache entries become stale when source documents, prices, policies, prompts, tools or models change. Time-based expiry is useful but insufficient. A 24-hour TTL still permits a full day of incorrect answers after an urgent policy correction. Record dependencies when writing each entry: document hashes, catalogue version, prompt version, schema version and model release. When any dependency changes, mark linked entries invalid or move queries to a new versioned namespace.
Use short TTLs for volatile facts and longer ones for stable transformations. A shipping estimate tied to live carrier data may merit minutes; an explanation of a long-standing API concept could remain valid for weeks. Risky categories should support event-driven purges. A product recall, security incident or legal update should trigger targeted invalidation by topic, source ID and tenant. Maintain a kill switch that bypasses the cache globally or for selected intents when confidence in stored responses falls.
Caching failures as well as successes demands care. A temporary upstream timeout should not be reused for an hour, while a deterministic “unsupported file type” response may be safely cached if the file extension and policy version are part of the key. Avoid caching refusals without understanding why they occurred: a refusal caused by missing context can incorrectly block a later, better-specified request. Every stored item needs provenance, expiry and a reason it is considered reusable.
Implement the request path with observable guardrails
A robust path begins by classifying cache eligibility. Exclude prompts containing secrets, live-state queries, side-effecting actions and requests whose answer must be personalised. For eligible traffic, construct the canonical representation, generate its embedding, and search the correct namespace with exact metadata filters. Retrieve a small candidate set, often three to ten entries, then apply threshold rules, recency checks and dependency validation. If no candidate survives, call the language model and write the response only after content and safety checks pass.
Do not let cache lookup become a new latency bottleneck. Set a tight budget, such as 50 milliseconds at the 95th percentile, and fall back to normal generation if the vector service is slow. Cache embeddings for repeated canonical requests where appropriate, batch writes asynchronously and keep the full path resilient to index outages. A hit should return the stored answer plus internal metadata identifying the matched request, similarity score and entry age; this metadata need not be exposed to the user, but it is essential for debugging.
Log decisions without recording unnecessary personal data. Useful fields include eligibility outcome, namespace, threshold policy, candidate scores, accepted entry ID, generation cost avoided, latency saved and invalidation version. Sample matched request pairs for human review, especially near the acceptance boundary. Alerts should detect sudden jumps in hit rate, cross-tenant filter failures, stale-dependency attempts and changes in similarity distributions after an embedding-model update.
Measure savings against answer quality
A semantic cache is successful only if it improves unit economics without degrading outcomes. Track eligible-request rate, lookup success, accepted-hit rate, precision of hits, response latency, model tokens avoided and total infrastructure cost. Include embedding generation, vector storage, search traffic, moderation, reranking and engineering overhead. If a cached generation saves £0.015 but lookup and validation cost £0.002, 1 million accepted hits save roughly £13,000 before operational costs. That figure is meaningful only when incorrect reuse remains within the product’s risk tolerance.
Run a shadow evaluation before serving cached answers. Perform semantic lookup, record what would have been returned, but still generate a fresh response. Compare both answers with human reviewers and task-specific checks: SQL execution equivalence, citation support, policy compliance, factual freshness or successful support resolution. Then conduct an A/B test with a small traffic slice. Monitor complaint rates, escalation rates, task completion and repeat queries alongside cost and latency. Users often reveal a bad cache hit by immediately rephrasing the same question.
Review false positives and false negatives separately. False positives indicate unsafe reuse and may require higher thresholds, better metadata filters or narrower eligibility. False negatives leave savings unrealised and may justify improved canonicalisation or domain-specific embeddings. Segment results by intent, language, tenant size and request age; an aggregate 25 per cent hit rate can conceal a 60 per cent rate for FAQs and near-zero value for personalised workflows. Expand semantic caching where precision is proven, and disable it where validation costs or error consequences outweigh the saved model call.
Comments (0)
Discussion is opening soon. Be the first to comment.