Build an LLM Judge That Knows When Human Review Is Required

Start with the decision, not the model

An LLM judge is not merely a model that assigns scores. It is a decision system that determines whether an output passes, fails or requires human review. That distinction matters because the operational cost of each error differs. Approving unsafe medical advice may cause harm; rejecting a good product description may only add editing time. Before selecting a model, define the decision, the affected users and the consequences of false acceptance and false rejection. A useful specification states the content type, evaluation unit, permitted evidence and required action for every verdict.

Translate those consequences into measurable targets. A customer-support team might require 99.5% recall for outputs containing fabricated refund policies, while accepting 92% precision because reviewers can absorb some false alarms. A marketing workflow may prioritise efficiency instead, automatically approving low-risk copy when the estimated probability of a pass exceeds 0.97. Report automation rate, severe-error recall, reviewer load and appeal overturns separately. A single accuracy figure conceals whether the judge is safe, useful or economically viable.

Build rubrics around observable evidence

Strong rubrics describe observable properties rather than impressions. “Good answer” invites inconsistent judgement; “states the return window accurately, identifies exceptions and does not invent fees” gives the judge evidence to inspect. Separate criteria such as factuality, instruction compliance, completeness, tone, safety and citation support. Use anchored scales: a factuality score of 4 might mean every material claim is supported, 2 might mean one consequential claim is unsupported, and 0 might mean the central recommendation is false.

Avoid forcing every criterion into one weighted average. A response scoring 5 for style and 0 for safety should not pass because presentation compensates for risk. Define hard gates for non-negotiable failures, then calculate a composite score for softer qualities. For example, require zero prohibited claims, factuality of at least 3 out of 4, and an overall weighted score above 80. Include counterexamples and boundary cases in the rubric: quoting a dangerous statement to refute it is different from endorsing it, while omitting a mandatory disclaimer is different from using an unfashionable tone.

Rubric development needs domain specialists, not only prompt engineers. Ask policy owners and frontline reviewers to label 100–300 representative outputs, explain disagreements and rewrite criteria that produce recurring confusion. Preserve those adjudicated cases as a versioned reference set. If experts cannot agree after consulting the same policy, the problem is probably an underspecified rule rather than an incapable judge.

Design the judge to expose its evidence

Require structured output that can be validated mechanically. A practical schema includes criterion scores, cited spans, policy references, an overall verdict, confidence and a review reason. Evidence should point to exact claims in the candidate response and, where applicable, the source document that contradicts or supports them. Reject malformed evaluations and retry under controlled limits. Free-form essays are harder to audit, compare and route, and they allow confident language to masquerade as reliable analysis.

Give the judge only the context necessary for the task. When checking a support answer, provide the customer question, approved policy passages and the proposed response, with clear delimiters and instructions that candidate text is untrusted data. This reduces prompt-injection risk and prevents the model from relying on general knowledge where company policy should govern. For long documents, retrieval quality becomes part of evaluation quality: if the decisive clause is absent, the judge should return “insufficient evidence”, not improvise.

Use multiple passes when stakes justify the cost. One pass can extract claims, another can verify them against evidence, and a final deterministic layer can apply routing rules. At £0.012 per evaluation, a three-pass design may cost more than a single £0.004 call, but it can be economical if it cuts human review from 60% to 20%. Measure latency and token cost against avoided review time, not against model expenditure in isolation.

Calibrate confidence instead of trusting self-reported certainty

A model’s stated confidence is not automatically a probability. It may report 95% confidence on cases that are correct only 78% of the time. Calibrate on held-out, human-adjudicated examples drawn from production. Useful signals include score margins, agreement across repeated judgements, rubric-specific uncertainty, evidence availability and disagreement between independent judges. Fit a calibration method such as isotonic regression or Platt scaling, then test reliability by confidence band: among decisions labelled 0.80–0.90, roughly 80–90% should be correct.

Set thresholds from risk and capacity. Suppose 10,000 outputs arrive daily and reviewers can inspect 1,200. An operating policy might auto-pass when calibrated pass probability is at least 0.98, auto-fail below 0.10, and send the remainder to review. If that produces 1,700 referrals, thresholds must change, reviewer capacity must increase or upstream quality must improve. Never quietly discard excess cases; use priority queues based on severity, uncertainty and exposure.

Thresholds should vary by criterion. A 0.90 probability may be adequate for detecting minor style defects but unacceptable for approving legal claims. Track selective accuracy: how accurate is the system on the subset it chooses to automate? Coverage should rise only while error remains within tolerance. A judge that automates 40% of cases at 99.7% accuracy may be more valuable than one automating 85% at 94%, particularly when errors are asymmetric.

Test for judge bias and systematic blind spots

LLM judges inherit preferences from training and prompting. They may favour longer answers, polished prose, familiar dialects, outputs from their own model family or responses placed first in a comparison. Detect these effects with controlled perturbations. Swap candidate order, remove brand names, vary length without changing substance, paraphrase in different registers and compare semantically equivalent answers using British, American and non-native English. Material score changes reveal sensitivity unrelated to the rubric.

Create slice-based reports rather than relying on aggregate agreement. Measure false-pass and false-fail rates by language, topic, response length, customer segment, model source and policy category. Overall agreement of 93% can conceal a 25% false-fail rate for concise answers or weak performance on disability-related requests. Minimum slice sizes and confidence intervals are essential; a dramatic percentage based on eight examples should trigger investigation, not a sweeping policy change.

Also test position bias and model-family affinity in pairwise evaluations. Run each comparison twice with reversed order and mark inconsistent outcomes for review. Where feasible, use a judge from a different provider or architecture than the system being assessed, then compare results with an expert panel. No judge is neutral, but biases can be measured, bounded and incorporated into routing.

Route ambiguity to qualified people

Human review should be a designed state, not an exception labelled “model failure”. Route cases when evidence is missing, judges disagree, calibrated confidence falls between thresholds, a hard-risk criterion is triggered or the input belongs to a poorly validated slice. Some cases require specific expertise: legal claims should go to legal reviewers, clinical content to clinicians and multilingual nuance to fluent speakers. Sending every ambiguous case to a general queue merely transfers uncertainty without resolving it.

Give reviewers the candidate output, relevant source evidence, rubric criterion, judge scores and the precise reason for escalation. Do not anchor them too strongly with a prominent model verdict; present evidence first or blind the initial decision where independence matters. Capture their ruling, edits, rationale, time spent and whether the case exposed a policy gap. A three-way choice—pass, fail or policy unclear—is more informative than forcing binary labels.

Use service levels based on potential harm. A suspected safety violation in a public-facing response may require review within 15 minutes, while a borderline tone issue can wait 24 hours. Monitor queue age, reviewer agreement and overturn rate by escalation reason. If reviewers overturn 45% of “unsupported claim” referrals, the rubric, retrieval system or calibration layer needs attention.

Validate in production and govern change

Before automation, run the judge in shadow mode on live traffic for two to four weeks. Compare its decisions with independent human labels, including every high-risk case and a random sample of apparent passes. Estimate false acceptance with appropriate uncertainty; observing zero severe errors in 500 cases does not prove zero risk. Red-team prompt injection, multilingual inputs, adversarial verbosity, conflicting sources and near-duplicate policies. Test failure modes such as timeouts, invalid JSON and retrieval outages as rigorously as semantic accuracy.

Once deployed, sample auto-passes and auto-fails continuously. Track policy-weighted error, automation coverage, reviewer load, latency, cost, drift and appeal outcomes. Use control charts or weekly confidence intervals to distinguish noise from deterioration. A sudden rise in review referrals may indicate a new input distribution, a changed source document or a provider model update. Maintain a kill switch that can move affected traffic to human review without requiring a software release.

Version the rubric, prompt, model, retrieval index, calibration set and thresholds together. Any material change should pass regression tests on adjudicated examples, bias slices and recent production cases. Keep decision logs sufficient for audit without retaining sensitive data longer than necessary. The strongest LLM judge is not the one that sounds most certain; it is the one whose limits are measured, whose uncertain cases reach the right experts and whose operating policy changes as evidence accumulates.