Inside AI Memory: Choosing Between Context, Retrieval, and State

AI memory is not one system

When an AI assistant appears to remember, several mechanisms may be working behind the interface. The current conversation can be held in a model’s context window; older material can be fetched from a search index; application facts can live in a database; and user preferences can be stored in a profile. These mechanisms differ in cost, latency, reliability and governance. Calling all of them “memory” obscures the engineering choices that determine whether an assistant recalls the right fact, invents one, or exposes information it should not see.

A dependable architecture starts by classifying what must persist. A customer’s latest message may need to survive for minutes. A product manual may remain useful for three years. An order status can change every hour, while a preference such as “use British spelling” may remain stable until explicitly edited. Each item belongs in a different layer. The central design question is therefore not how to give a model more memory, but where each fact should live, how it should be retrieved, and which source wins when records conflict.

Context is fast, flexible and temporary

Context is the information supplied directly to the model for one inference: system instructions, conversation turns, tool results and supporting documents. It is the closest equivalent to working memory. Because the model can attend to this material without making another retrieval call, context is usually the simplest way to preserve conversational continuity. A support assistant can keep the last six exchanges, the active ticket summary and the latest tool response together, allowing it to answer “When will it arrive?” without asking which parcel the customer means.

Large context windows do not remove the need for discipline. A nominal 128,000-token window must accommodate instructions, user messages, retrieved evidence and the output itself. Sending 80,000 irrelevant tokens increases input cost and can reduce accuracy as salient details compete with noise. Repeatedly appending entire transcripts also creates latency and privacy risks. Teams should trim boilerplate, retain recent turns, and summarise older discussion into explicit decisions, unresolved questions and named entities. Summaries must be regenerated carefully: one incorrect compression can contaminate every later response.

Context is best for immediate intent, temporary constraints and material needed verbatim. It is poor as a durable record because it disappears when the session ends, may be truncated, and cannot reliably represent concurrent updates. If a user changes a delivery address, the new address should not exist only in the chat history. The assistant should write it through an authorised application workflow, then return the confirmed database value to context.

Retrieval provides scale, not certainty

Retrieval-augmented generation stores documents outside the model, converts them into searchable representations and selects relevant passages at request time. This allows an assistant to work across millions of pages without placing the whole corpus in every prompt. A manufacturer might divide 10,000 service manuals into passages of 400 to 800 tokens, attach metadata for model, year and market, and retrieve the top five passages for a technician’s question. The economics are compelling: index once, search cheaply, and pay model input costs only for selected evidence.

The weakness lies in selection. Semantic similarity can return text that sounds relevant but applies to the wrong product generation. Keyword search can miss paraphrases. Hybrid retrieval, combining vector similarity with lexical ranking, often performs better than either method alone. Metadata filters are equally important: a query about a 2024 UK vehicle should not retrieve a 2019 US wiring diagram merely because the component names match. Reranking the first 30 candidates into the best five can improve precision, although it adds another model call and perhaps 100 to 300 milliseconds of latency.

Retrieval should carry provenance. Passages need document identifiers, publication dates, access controls and, where possible, effective dates. The model should cite those sources and decline to make strong claims when evidence is absent or contradictory. Evaluation must test retrieval separately from answer generation. If the correct passage appears in the top five only 72 per cent of the time, polishing the final prompt cannot overcome the missing 28 per cent.

Structured state is the source of truth

Structured state covers facts represented as explicit fields, records and transitions: account balance, subscription tier, shipment status, workflow stage or approval decision. Unlike prose, these values have schemas, validation rules and ownership. An order can be “dispatched” or “delivered” according to a defined system, not according to the model’s interpretation of a conversation. When an assistant answers operational questions or performs actions, structured state should generally outrank both remembered dialogue and retrieved documents.

Consider a sales agent handling a renewal. The transcript says the customer was offered a 15 per cent discount, the profile says they prefer annual billing, and the billing platform records that the offer expired yesterday. The model must not infer that the discount still applies. It should query the billing system, receive an expiry timestamp and use that current value. Writes require even tighter controls: validate arguments, request confirmation for consequential changes, use idempotency keys to prevent duplicate transactions, and record who or what initiated the action.

Structured state costs more to design because engineers must define schemas, APIs and failure behaviour. Yet that investment buys determinism. A database query can return an exact quantity and update time; a vector search returns candidates. Good systems also distinguish observation from authority. A customer may state, “I paid last Friday”, but until a payment system confirms the transaction, that claim remains conversational evidence rather than settled account state.

User profiles should be sparse and inspectable

A user profile is durable personalisation: language, accessibility needs, preferred units, communication style, recurring goals and consent settings. Profiles can make assistants more useful without requiring users to repeat themselves. A travel service might remember that a customer prefers rail journeys under four hours and needs step-free access. These fields can guide ranking and presentation while leaving live availability and prices to transactional systems.

The danger is turning every utterance into a permanent trait. A user asking for a cheap hotel once does not necessarily prefer budget accommodation. Profile updates should meet a confidence threshold, be based on repeated behaviour or explicit statements, and include provenance and timestamps. Stable facts and inferred preferences should be stored separately. “User selected kilometres” is stronger than “User may dislike imperial units”. Inferred fields should decay or be reviewed; a 90-day expiry may suit shopping preferences, while an accessibility requirement should persist until changed.

Profiles must be visible and controllable. Users should be able to inspect, correct and delete remembered preferences, and organisations should minimise sensitive data. Storing medical, political or financial inferences merely because they could improve personalisation creates disproportionate risk. Access should be scoped by product and purpose: a support bot may need a customer’s language preference, but not the detailed browsing history collected by a marketing system.

A memory router decides what belongs where

Mature AI products use routing rules rather than a single memory store. The system classifies incoming information by lifespan, authority, sensitivity and update frequency. Ephemeral instructions remain in context. Unstructured reference material goes into retrieval. Operational facts enter structured systems through validated tools. Durable preferences go into profiles after consent and confidence checks. This division can be implemented as policy code, a classifier, or both, but high-risk decisions should not rely solely on a probabilistic model.

At response time, the router assembles a bounded memory package. For example, a banking assistant may receive the last four conversation turns, a 200-token summary, three policy excerpts, the live account balance and two approved profile fields. Precedence should be explicit: current system policy outranks old policy documents; live ledger data outranks user recollection; the user’s latest explicit preference outranks an older inferred profile. When conflicts cannot be resolved, the assistant should name the discrepancy rather than silently blending incompatible facts.

Memory writes need their own pipeline. Extract a candidate fact, classify it, check permissions, validate it against a schema, deduplicate it, attach provenance and decide its retention period. A statement such as “Call me Sam” may become a profile preference immediately. “My company has 2,000 employees” may be stored as an unverified account note, not overwritten into a verified corporate record. This distinction prevents conversational convenience from corrupting authoritative data.

Dependability is measured through failure modes

Memory quality cannot be reduced to whether users feel recognised. Teams should measure retrieval recall, answer faithfulness, state freshness, write accuracy, privacy violations and latency. A practical test set might contain 500 scenarios across fresh sessions, long conversations, contradictory documents and revoked permissions. Useful targets could include 95 per cent top-five retrieval recall for supported questions, zero unauthorised cross-tenant results, and 99.9 per cent successful propagation of confirmed profile edits within one minute.

Adversarial testing is essential. Evaluators should insert outdated manuals, similar customer names, malicious instructions inside retrieved documents and profile fields that conflict with the latest request. They should also simulate tool timeouts and stale caches. If the order API is unavailable, the assistant must say that it cannot verify status, not substitute yesterday’s transcript. Logs should show which memory items were read, which were written, their provenance and the policy that authorised access, while avoiding unnecessary capture of raw sensitive content.

The strongest design is deliberately hybrid. Context preserves the thread, retrieval opens a large knowledge base, structured state anchors current facts, and profiles provide controlled personalisation. Dependability comes from boundaries between those layers: narrow permissions, explicit precedence, traceable sources, retention limits and rigorous evaluation. An assistant does not become trustworthy by remembering everything. It becomes trustworthy by remembering the right information in the right form, and by knowing when memory is not evidence.