RAG Is Not a Database: The Architecture Mistakes Teams Keep Making

RAG retrieves evidence; it does not manage truth

Retrieval-augmented generation is often sold as a simple upgrade path: connect a language model to company documents, embed the content, and let employees ask questions. That description hides the real system. RAG is not a database, a content management platform or an access-control layer. It is an application pattern in which a retrieval service selects evidence and a generative model produces an answer from it. If the underlying information is duplicated, stale, poorly scoped or inaccessible, the model cannot repair it. It may instead convert those defects into fluent, authoritative prose.

Consider a support organisation with 80,000 knowledge-base articles, product manuals and resolved tickets. If three versions of a cancellation policy remain indexed, retrieval may surface the obsolete version because its wording more closely resembles the user’s question. The model is then doing exactly what it was designed to do: synthesising the supplied context. The failure occurred before generation, in source governance, indexing and ranking. Treating the model as the system of record obscures ownership and makes debugging needlessly subjective.

A sound architecture therefore separates responsibilities. Systems of record own transactions and canonical facts. Content platforms own documents, versions and workflow. Identity infrastructure owns users, groups and entitlements. Search infrastructure owns indexing and retrieval. The model interprets selected evidence and communicates it. That division may appear less magical than “chat with your data”, but it creates testable boundaries, operational controls and a credible route from prototype to production.

A vector index is a projection, not a source of record

Teams routinely mistake the vector store for the database because it contains copies of their content. Yet an embedding is a lossy numerical projection optimised for similarity, not an authoritative representation of a document. It cannot reliably preserve every date, identifier, exception or relationship. Even the stored text chunks are usually derivatives: headings removed, tables flattened, pages split and metadata transformed. Rebuilding the index should always be possible from canonical sources; editing it directly should not be part of normal operations.

This distinction matters when records change. Suppose an insurer updates a policy from a £5,000 to a £2,500 excess. The transactional or content system records the new value, author, approval time and effective date. A vector index may still contain the old passage until ingestion runs, while caches may retain it longer. Without lineage fields such as source ID, version, checksum, effective date and ingestion timestamp, operators cannot determine which representation produced an answer. A citation that merely links to the document homepage is not enough.

Use the vector index as a disposable serving layer. Keep canonical identifiers and source URLs with every chunk, maintain an ingestion ledger, and make deletions reproducible. Where questions depend on exact values, joins or current status, query structured systems instead of asking semantic search to infer them. “What does our travel policy say about meals?” may suit document retrieval. “How much budget remains on project 4187?” belongs in a governed database query or API call, followed by controlled presentation through the model.

Indexing quality determines the ceiling

A RAG stack cannot retrieve information it has failed to represent. Indexing is therefore an editorial and engineering discipline, not a one-off embedding job. Chunk size, overlap, document structure, metadata and parsing all influence recall. A fixed 1,000-token splitter may cut a heading from its qualifying paragraph, separate a table row from its column labels, or merge unrelated clauses. Smaller chunks can improve precision but lose context; larger chunks preserve context but consume the model’s window and introduce distracting material.

The right unit is usually semantic rather than arbitrary. Product documentation may be split by heading hierarchy, legal contracts by clauses, support cases by issue and resolution, and transcripts by speaker turns or topics. Tables often need a parallel textual representation that repeats headers for each row. Images and scanned PDFs require optical character recognition, with confidence thresholds and review for high-risk sources. If 7 per cent of pages fail parsing, a polished chatbot does not make the missing knowledge disappear.

Teams should measure indexing separately from answer generation. Build a representative set of queries and identify the exact passages that should be retrieved. Track recall at k, precision at k, duplicate rate and the proportion of chunks with valid lineage. If the required passage appears in the top five results only 72 per cent of the time, prompt changes cannot raise grounded-answer performance beyond that constraint. Inspect misses by document type, language, age and parser. This is search relevance work, and it requires the same rigour search teams have applied for decades.

Permissions must travel with every chunk

The most dangerous RAG error is not a weak answer but an unauthorised one. Copying files from SharePoint, Google Drive or an internal wiki into a common index can strip away folder permissions, group membership and document-level restrictions. A model asked about “planned redundancies” may retrieve an HR memorandum that the user could never open in the source system. Telling the prompt not to reveal confidential information offers no protection; once restricted text enters the context window, disclosure has already become possible.

Access control must operate before retrieval results reach the model. Each indexed object needs permission metadata tied to stable identities or groups, and the retrieval service must filter candidates using the requesting user’s current entitlements. Post-filtering the top ten vector matches is often insufficient: if nine are forbidden, the system may return one weak result even though better authorised results exist deeper in the index. Pre-filtering, partitioned indexes or oversampling with strict filtering each carry performance and operational trade-offs, but all are preferable to prompt-based security.

Permissions also change. Employees leave teams, deals close and investigation folders become restricted. Synchronisation must handle grants and revocations, with revocation usually requiring a tighter service-level objective than ordinary content updates. Log the user identity, policy decision, source IDs and returned chunks without indiscriminately storing sensitive prompts. Then test adversarially: use accounts from different departments, attempt indirect questions, and verify citations cannot expose titles or snippets from forbidden documents. Security belongs in retrieval architecture, not in model etiquette.

Freshness requires a lifecycle, not a nightly upload

Many pilots rely on a nightly batch that scans documents and regenerates embeddings. That may be adequate for an employee handbook updated quarterly, but it is reckless for stock levels, incident procedures or prices. Freshness is a business requirement expressed as time: five minutes for an outage runbook, one hour for sales collateral, one day for historical research. Without explicit targets, every source inherits the same pipeline despite radically different risk.

A robust ingestion lifecycle detects creations, edits, moves and deletions. Event-driven connectors can reduce lag, while periodic reconciliation catches missed events. Checksums prevent unnecessary embedding work, and versioned indexes enable safe rollout and rollback. Deletion deserves particular attention: removing a source file must remove its chunks, cached results and derived summaries. Otherwise, “right to be forgotten” requests and policy withdrawals become fiction. For frequently changing structured data, retrieval-time API calls are often safer and cheaper than continuous re-embedding.

Freshness should also influence ranking and presentation. A superseded technical bulletin may remain useful for customers running an older product version, while a current bulletin should dominate for the latest release. Store effective dates, product versions and supersession relationships, then use them as filters or ranking features. Display the source date in citations and let the model state when evidence conflicts. Silence about age encourages users to assume that every retrieved passage is current, an assumption the architecture may not support.

Query design is more than embedding a sentence

Users do not naturally phrase questions in the form best suited to retrieval. They use pronouns, abbreviations, misspellings and conversational follow-ups. “What about contractors?” may depend entirely on the preceding question about parental leave. Sending that fragment directly to a vector index discards necessary context. Query rewriting can produce a self-contained search request, but it must preserve constraints rather than invent them. The original question should remain available for auditing and answer generation.

Semantic similarity alone is also a poor fit for many enterprise queries. Product codes, error messages, names and statutory references often require exact lexical matching. Hybrid retrieval combines dense vectors with methods such as BM25, then reranks candidates using a cross-encoder or model. A query containing “E1047” should reward an exact match; a query asking how to restore access after repeated login failures benefits from semantic matching. Metadata filters for region, product, date and document status can narrow the search space before ranking.

Complex questions may need decomposition. “Compare our 2025 UK and French parental-leave policies” contains at least two retrieval tasks plus a comparison step. An orchestrator can retrieve each jurisdiction separately, verify that both policies are current, and ask the model to compare only cited evidence. The trade-off is latency and cost: three retrieval calls and a reranker may turn a 600-millisecond search into a three-second answer. Teams should reserve elaborate pipelines for questions that justify them rather than applying agentic machinery to every request.

Evaluation must isolate retrieval from generation

End-to-end demonstrations conceal where failures originate. A useful evaluation programme separates retrieval quality, context quality and answer quality. For retrieval, measure whether authoritative passages appear near the top. For context, measure duplication, contradiction, permission compliance and token efficiency. For generation, assess factual consistency with supplied evidence, completeness, citation accuracy and appropriate refusal. Human judgement remains valuable, but repeatable labelled tests are essential for comparing releases.

Start with 200 to 500 questions drawn from real workflows, not only questions written by the project team. Include unanswerable requests, ambiguous wording, outdated terminology, permission boundaries and questions requiring exact figures. Record the expected sources and acceptable answer attributes. Then run controlled experiments: change chunking without changing the model, change ranking without changing the prompt, and change generation settings without rebuilding the index. Otherwise, an apparent improvement cannot be attributed to any component.

Production metrics should include retrieval latency, answer latency, empty-result rate, citation-open rate, user correction rate and the age distribution of returned sources. Sample traces for expert review, with appropriate privacy controls. A thumbs-up metric alone is misleading: users often reward concise confidence even when the answer is wrong. In regulated settings, a correctly refused answer may be more valuable than a plausible response assembled from weak evidence. Evaluation should reflect the cost of each failure, not merely average satisfaction.

Build a governed retrieval system, then add generation

The durable architecture begins with an inventory of sources and decisions about authority. For each source, define its owner, sensitivity, update pattern, canonical identifiers and deletion process. Establish an ingestion pipeline that preserves structure, lineage and permissions. Use multiple retrieval methods where the corpus demands them, and route structured questions to structured tools. Only after the system can reliably return the right evidence to the right user should a language model be placed on top.

The model layer should be deliberately constrained. Provide a bounded set of passages, require citations at claim level where risk warrants it, and instruct the system to distinguish evidence from inference. If sources conflict or do not support an answer, the interface should say so and offer the underlying documents. Caching can reduce cost, but cache keys must account for identity, permissions, source versions and query parameters; a shared answer cache can otherwise become both a freshness bug and a data leak.

This approach costs more than a weekend prototype because it includes search engineering, identity integration, data governance and observability. It also scales more honestly. Teams can replace the embedding model, vector engine or generator without redefining truth or rebuilding access control from prompts. The decisive shift is conceptual: RAG is a retrieval and synthesis layer over governed information systems. Once that boundary is respected, failures become measurable engineering problems rather than mysterious episodes of model behaviour.