How to Measure AI Answer Quality When No Gold Standard Exists

Stop treating correctness as a single observable fact

Many AI systems operate where definitive answers are scarce, delayed or inherently contested. A support assistant may draft a tactful response without there being one perfect formulation. A research tool may summarise an emerging field whose evidence is incomplete. A coding agent may produce several implementations that all pass the available tests. In these settings, conventional accuracy is not merely difficult to calculate; it is the wrong abstraction. Quality is multidimensional, combining factuality, relevance, completeness, reasoning, safety, style and usefulness for a particular user.

The first measurement task is therefore to define the decision the answer must support. A legal summariser used for case triage should be judged heavily on faithful citation and omission risk. A shopping assistant may be judged on constraint satisfaction, recommendation diversity and whether users later return products. The same fluent answer can be excellent in one context and dangerous in another. Teams should write an explicit quality model before building a benchmark, assigning dimensions and weights tied to product risk rather than adopting generic notions such as ‘helpfulness’.

A practical scorecard might allocate 30 per cent to factual support, 25 per cent to task completion, 20 per cent to relevance, 15 per cent to clarity and 10 per cent to safety. These weights are hypotheses, not laws. They should be tested against expert judgement, user outcomes and known failure costs. If an unsupported medical claim is ten times more costly than an awkward sentence, the evaluation system must reflect that asymmetry instead of averaging both defects into a deceptively respectable score.

Build expert rubrics around observable evidence

A useful rubric converts broad ideals into behaviours that reviewers can identify consistently. ‘Accurate’ is too vague. ‘Every material claim is supported by the supplied documents or a verifiable source’ is assessable. ‘Complete’ becomes ‘addresses all four constraints in the user request’. For each dimension, define anchored levels. A four-point factual-support scale might range from zero for material fabrication, through one for major unsupported claims and two for minor unsupported details, to three for full support with correct attribution.

Anchors should include real examples from the product, especially borderline cases. Suppose a financial assistant says a fund ‘usually outperforms during inflation’. Reviewers need guidance on whether that is a factual claim requiring a time period, benchmark and source, or acceptable shorthand. Without examples, experts silently apply different standards. Ten reviewers can then produce ten plausible but incompatible labels, creating the appearance of model volatility when the real problem is an underspecified rubric.

Calibration should precede production scoring. Ask reviewers to rate the same 30 to 50 answers independently, discuss disagreements and revise the guide. Measure inter-rater reliability, but do not worship a single statistic: Krippendorff’s alpha of 0.75 may be adequate for exploratory comparisons yet insufficient for safety-critical deployment. Recalibrate after model, policy or domain changes. Expert evaluation is not a one-off labelling exercise; it is a maintained measurement instrument.

Use pairwise comparisons when absolute scores wobble

People often struggle to assign an isolated answer a score of 73 out of 100, but can reliably decide whether answer A or answer B is better. Pairwise comparison reduces cognitive load and exposes meaningful product differences. Present two responses to the same prompt in random order, hide model identity and ask which better satisfies the rubric. Include a tie option so reviewers are not forced to invent distinctions.

Pairwise data can rank several systems through methods such as Bradley–Terry modelling or Elo-style updates. If a candidate wins 62 per cent of 800 blinded comparisons against the current production model, the result is more informative than a two-point rise on a subjective ten-point scale. Confidence intervals still matter: a 52 per cent win rate from 50 comparisons is weak evidence, while the same rate across 10,000 well-sampled comparisons may reveal a small but commercially valuable improvement.

The tradeoff is diagnostic depth. A preference identifies a winner but may not explain why it won. Add structured reason codes such as ‘better evidence’, ‘missed constraint’, ‘clearer’, ‘safer’ and ‘more concise’, then audit free-text comments on a subset. Control for length bias, verbosity and position effects; reviewers frequently prefer longer answers even when extra text adds no value. Pairwise evaluation works best alongside rubrics, not as a substitute for them.

Treat disagreement as data, not labelling noise

Reviewer disagreement can reveal ambiguity in the prompt, contested domain knowledge or legitimate differences in user preference. Consider an assistant asked for ‘the best city for a remote worker’. One reviewer may prioritise cost, another internet reliability and a third visa rules. Averaging their scores produces a number but erases the reason no single gold answer exists. The disagreement is itself a product insight: the assistant should ask a clarifying question or state its assumptions.

Track disagreement by dimension, prompt type and reviewer cohort. High variance on tone may be tolerable; high variance on whether a claim is supported signals a serious rubric or evidence problem. Examine cases where experts split 50–50, where confidence is low, and where specialists systematically diverge from generalists. If clinicians rate a health answer as unsafe while ordinary users find it reassuring, optimising for aggregate preference would be reckless.

Do not automatically adjudicate every split into one official label. Preserve distributions where uncertainty is genuine. A response rated safe by seven experts and unsafe by three is different from unanimous approval, even if both receive the same majority label. Models and release gates can incorporate that uncertainty through conservative thresholds, risk-weighted scores or escalation rules. This produces a more honest evaluation than forcing complex judgement into artificial certainty.

Connect offline judgement to behavioural signals

Expert ratings estimate quality; user behaviour tests whether that quality matters in practice. Useful signals include task completion, follow-up questions, reformulation rates, citation clicks, copy actions, abandonment, support escalation and downstream corrections. For a coding assistant, the strongest signal may be whether suggested code compiles and survives tests. For a service agent, it may be resolution without reopening the case within seven days.

Behavioural metrics are powerful but confounded. A high click-through rate can reward sensational recommendations. Short sessions may indicate either rapid success or immediate abandonment. Users can accept confident misinformation because they cannot detect it. Teams should define a causal story for each metric and validate it against expert-reviewed samples. If answers receiving high factuality scores also reduce correction rates by 18 per cent, that relationship supports the metric; if engagement rises while verified error rates worsen, engagement is a poor quality proxy.

Online experiments should use guardrails. Compare variants on primary outcomes while monitoring harmful claims, complaints, latency and cost. A model that improves successful task completion from 68 to 72 per cent but doubles serious factual errors from 0.4 to 0.8 per cent may be unacceptable in a regulated product. Segment results by task difficulty and user group, because aggregate gains can conceal regressions for rare languages, complex queries or vulnerable users.

Sample where failures hide, not merely where traffic is

Random sampling estimates average performance, but average performance can conceal severe weaknesses. If 95 per cent of requests are simple rewrites, a representative sample may barely test numerical reasoning, multilingual safety or citation fidelity. Evaluation portfolios should combine traffic-weighted samples with targeted strata: rare intents, high-stakes domains, long contexts, adversarial prompts, new features and cases previously associated with incidents.

A sensible monthly programme might devote 50 per cent of reviews to representative production traffic, 25 per cent to known risk categories, 15 per cent to recent failures and 10 per cent to exploratory edge cases. Oversampling distorts prevalence, so report both raw stratum scores and reweighted overall estimates. A system may achieve 91 per cent acceptable answers on traffic-weighted data yet only 63 per cent on complex document comparisons; both figures are necessary for an honest release decision.

Active sampling can make limited review budgets more effective. Prioritise answers where models disagree, automated checks flag unsupported claims, confidence is low or user behaviour deviates sharply from normal patterns. However, maintain a random control sample to detect unknown unknowns. Sampling only what existing detectors consider suspicious creates a closed loop in which novel failure modes remain invisible.

Create an evaluation system that can survive model change

Quality measurement should operate as a layered system rather than a single leaderboard. Use deterministic checks for format, links, calculations and policy constraints; expert rubrics for nuanced judgement; pairwise tests for relative progress; behavioural experiments for user impact; and targeted audits for low-frequency risks. Each layer answers a different question. Agreement across layers is strong evidence, while contradiction is a prompt to investigate rather than choose the most flattering metric.

Maintain a versioned evaluation set with protected holdouts, rotating production samples and a living challenge suite. Static benchmarks become contaminated as prompts leak into training, teams tune repeatedly against them or the product’s traffic changes. Retire obsolete cases, but retain a stable core for trend analysis. Record prompt versions, model settings, retrieval results, reviewer guidance and evaluation dates so a score can be reproduced rather than treated as an unexplained historical fact.

Release decisions need explicit thresholds. A candidate might require a statistically credible pairwise win, no more than a one-point regression on any critical rubric dimension, zero newly discovered severe safety failures in a designated audit, and acceptable latency and cost. When no gold standard exists, disciplined triangulation is the standard: multiple imperfect measurements, transparent uncertainty and rules tied to real consequences. The aim is not to manufacture certainty, but to make better decisions despite its absence.