The Practical Guide to Versioning Prompts, Models, and Evaluations

Version the whole behaviour, not just the prompt

An AI feature is not defined by a prompt alone. Its behaviour emerges from a linked set of production artefacts: system and user prompts, model identifier, sampling settings, tool schemas, retrieval configuration, safety rules, output parser, evaluation set and application code. Change any one of them and the result may shift. A team that records only “prompt v12” cannot reliably explain why extraction accuracy fell after a release, because v12 may have run against two model snapshots, three search indexes and an undocumented temperature change.

Treat each deployable configuration as an immutable release bundle. A useful manifest might contain prompt commit 7f31c2, model vendor/model-2026-05-14, temperature 0.2, maximum output of 1,200 tokens, tool-schema version 4, retrieval index 2026-06-01, evaluator pack 18 and application commit 91ab4e. Give that bundle a release identifier such as assistant-2.7.0, then store it beside the code. The manifest becomes the unit of review, deployment, monitoring and rollback.

This discipline separates source versions from release versions. A prompt can have dozens of commits without each becoming a production release. Conversely, one release may change only a model setting while leaving the prompt untouched. Semantic versioning can help: reserve major increments for contract-breaking changes, minor increments for intended capability changes and patches for compatible fixes. The precise convention matters less than ensuring every production response can be traced to one complete bundle.

Store prompts as reviewable source files

Prompts should live in source control as plain text or structured files, not solely inside a vendor console or an employee’s notebook. Separate stable instructions, examples, policy text and runtime variables so reviewers can see what changed. Templates should declare expected inputs and outputs, including types, required fields and failure behaviour. If a prompt references “the policy below”, the policy version must be pinned rather than fetched from a mutable document at runtime.

Write commit messages around behavioural intent. “Reduce unsupported refund promises by requiring policy citations” is useful; “prompt tweaks” is not. Pull requests should include the diff, affected use cases, evaluation results, estimated cost and known regressions. Two reviewers are sensible for high-risk workflows, with at least one domain owner checking the underlying policy rather than merely the wording. Prompt prose can look harmless while silently changing a legal or operational rule.

Avoid embedding secrets, customer data or vendor credentials in examples. Use synthetic fixtures and stable placeholders such as {{customer_tier}}. Where prompts are assembled dynamically, log the template version and a cryptographic hash of the rendered prompt, while applying appropriate redaction. The hash proves which content was sent without forcing sensitive material into a broadly accessible analytics store. Access controls and retention periods should match those applied to production data.

Pin models, parameters and provider behaviour

A model name such as “latest” is not a version. Providers can alter weights, routing, safety layers and context handling behind an alias, producing a behavioural release without a code change on your side. Use dated snapshots or explicit model revisions wherever the provider supports them. If snapshots are unavailable, record the alias, request time, API version, region and provider response metadata, then monitor for drift more aggressively.

Parameters belong in the release manifest. Temperature, top-p, seed, token limits, reasoning mode, response format and stop sequences can materially affect quality, latency and cost. Raising temperature from 0.1 to 0.8 may improve variety for campaign copy while damaging consistency in invoice extraction. Increasing a reasoning budget can lift a difficult benchmark by five percentage points but double median latency and add several pence per request. Those are product decisions, not implementation details.

Determinism should not be overstated. A fixed seed and temperature of zero may reduce variation, but distributed inference, provider updates and tool calls can still yield different outputs. Reproducibility therefore means reconstructing the conditions and obtaining statistically comparable behaviour, not necessarily reproducing every token. For critical tasks, run each test case three to five times and report pass-rate distributions rather than relying on one convenient sample.

Treat tools and retrieval as versioned dependencies

Tool use expands the versioning surface. A model may call get_order_status correctly under schema v3 and fail under v4 because “order_id” became “reference”. Version tool names, JSON schemas, descriptions, permissions, timeout rules and error responses. Keep backward-compatible endpoints during migration where feasible. A tool change should trigger contract tests before any end-to-end model evaluation begins.

Retrieval systems require the same rigour. Record the document corpus, chunking algorithm, embedding model, metadata filters, ranking method, reranker and top-k value. Changing chunks from 800 to 300 tokens can improve citation precision but remove context needed to interpret exceptions. Rebuilding an index with a new embedding model may alter results even when every source document is unchanged. Assign each index an immutable identifier and retain at least the versions needed to reproduce current and previous releases.

Mocks are useful but insufficient. They make tests fast and stable, yet they hide authentication failures, rate limits and malformed live responses. Use a layered suite: schema tests on every change, deterministic mocked scenarios in continuous integration, and a smaller set of scheduled tests against staging services. Capture tool-call arguments and results with sensitive fields redacted. When quality drops, these traces reveal whether the model reasoned badly or simply received different evidence.

Build evaluation sets that survive contact with production

An evaluation set is a production artefact, not a spreadsheet assembled once before launch. Give every item a stable identifier, input, expected properties, source, risk label and rationale. Include ordinary cases, edge cases, adversarial inputs and known failures. For a support assistant, a 600-case set might allocate 300 routine queries, 150 policy-sensitive cases, 100 tool failures and 50 injection attempts. Keep a separate holdout set to limit overfitting by repeated prompt edits.

Version the scoring logic alongside the cases. Exact match works for account numbers; it is poor for helpful explanations. Combine deterministic checks, such as valid JSON and correct citations, with calibrated rubric grading for dimensions such as factuality, completeness and tone. Model-based judges can scale review, but their model, prompt and settings must also be pinned. Validate judges against human ratings, particularly near release thresholds and for protected or regulated topics.

Evaluation data ages. Product policies change, users discover new failure modes and language distributions shift. Add production incidents as regression cases, but do not overwrite old expectations silently. If a return window changes from 30 to 45 days, create a new evaluation version and document the effective date. A monthly review can retire obsolete tests, rebalance categories and inspect leakage. The aim is a living benchmark with a clear history, not a permanently rising score on stale questions.

Set release gates around quality, cost and latency

A single average score conceals damaging regressions. Define gates by task and risk tier: for example, at least 98 per cent schema validity, 95 per cent citation support, zero critical policy violations across 200 high-risk cases, p95 latency below 4.5 seconds and mean model cost below £0.06 per completed request. Compare the candidate with the production baseline using identical cases and repeated runs. Report confidence intervals where outputs are variable.

Trade-offs should be explicit. A candidate may improve answer quality from 86 to 90 per cent while increasing p95 latency from 3.2 to 6.8 seconds and cost by 70 per cent. That could be justified for complex research requests but not for password-reset guidance. Route by task when possible rather than forcing one model configuration across the product. Release notes should state wins, losses and the population expected to benefit.

Automate gates in continuous integration, but retain human review for consequential changes. A practical pipeline validates files, runs unit and contract tests, executes a fast 100-case evaluation on pull requests, then runs the full suite before deployment. Store raw outputs, scores, evaluator versions and summaries as build artefacts. A release should fail because a named threshold was breached, not because somebody felt that several examples looked worse.

Roll out gradually and make rollback routine

Passing offline tests earns a candidate exposure, not trust. Begin with shadow traffic where outputs are generated but not shown, then move through a canary such as 1 per cent, 10 per cent and 50 per cent of eligible requests. Compare cohorts on task success, correction rate, escalation, user complaints, latency, cost and safety incidents. Keep assignment stable by user or conversation so one session does not alternate between incompatible behaviours.

Define rollback triggers before launch. Examples include a two-percentage-point drop in successful resolutions, a 25 per cent rise in tool errors, any critical data disclosure, or p95 latency exceeding the service objective for 15 minutes. Rollback should switch an alias or feature flag to the previous immutable bundle; it should not require rebuilding prompts or guessing old settings. Preserve compatible tool endpoints and indexes long enough to support that fallback.

Incident records should name the affected release, first detection time, traffic share, root cause, mitigation and new regression tests. Avoid blaming “the model” when the actual cause was a changed retrieval filter or parser. A concise provenance trail can reduce diagnosis from days to hours: response ID to release bundle, bundle to component versions, component versions to diffs and evaluation evidence. That is the operational value of versioning.

Create ownership, retention and a release ledger

Versioning fails when ownership is diffuse. Assign an owner to each prompt family, tool contract, retrieval index and evaluation pack, plus one release owner who approves the assembled bundle. Establish a change policy that distinguishes low-risk editorial adjustments from high-risk policy or permission changes. The latter should require domain approval, security review and stronger evaluation gates. Vendor-console edits should be restricted or automatically synchronised back to source control.

Maintain a searchable release ledger containing identifiers, dates, owners, manifests, evaluation summaries, approvals, rollout history and incidents. Retention should reflect risk: a marketing-copy assistant may need months of history, while a regulated decision-support system may require years. Store enough representative outputs to audit behaviour, but minimise personal data and respect deletion obligations. Provenance is valuable only if it is secure and intelligible.

The mature workflow is straightforward: propose a change, create a diff, assemble an immutable bundle, evaluate it against a versioned test set, review the trade-offs, canary it, monitor it and retain a one-action rollback. This adds process, but it removes a more expensive form of work: reconstructing undocumented production behaviour after quality shifts. Teams that version the whole system can improve faster because every experiment has evidence, every release has a history and every failure becomes a durable test.