How to Validate an LLM Judge

Asking a model to grade your model’s outputs takes about ten minutes to set up. Knowing whether that grader is right takes considerably longer, and most teams never do it. Ask how they know the judge is trustworthy and the room goes quiet [1].

That matters because an unvalidated judge does not produce a weak signal. It produces a confident, precise, wrong number, and every decision downstream inherits the error. A judge that systematically misses the failure you care about will show a clean dashboard right up until customers complain.

This guide covers how to build a judge worth trusting and how to measure whether you succeeded. It is part of our series on evals and LLMOps, and it assumes you have already done error analysis, because you cannot write a good judge prompt for a failure mode you have not yet seen in your own data [2].

Start by not building a judge

Three checks before you invest.

Is a code assertion enough? If a deterministic rule can catch the failure, write the rule. JSON parses, the citation exists, the tool call carried the right order ID, the date is formatted correctly. Assertions are free, instant, and never drift. Judges typically need 100 or so labeled examples per failure mode plus continuing maintenance, so they should be reserved for failures that genuinely require judgment and that you will iterate on repeatedly [3].

Is the failure just an unstated requirement? Teams regularly discover their system is not meeting a preference nobody ever wrote down. Fix the prompt and the failure disappears without any measurement infrastructure [2].

Is your feature scoped tightly enough to judge? The judge-building process reliably fails when the AI is overscoped, such as a chatbot that promises to do anything at all in a SaaS product. If no expert could say what a good answer looks like, no judge can either [2].

Step 1: Find the one person whose judgment counts

Pick a single domain expert who genuinely represents your users and make them the final authority on quality: a clinician for a health assistant, a lawyer for contract analysis, a support director for a helpdesk feature [2].

One expert rather than a committee, for a practical reason. Colleagues will disagree about what is acceptable, and averaging their opinions produces a standard nobody actually holds. A single authority keeps labels consistent, surfaces expectations the expert could not have articulated up front, and gives them ownership of the result [2]. Larger organizations spanning multiple domains sometimes need several annotators, and then you measure agreement between them with a chance-corrected metric such as Cohen’s kappa, resolve disagreements by amending the rubric rather than splitting the difference, and relabel affected examples [3].

Step 2: Build a dataset that covers your real variation

The labeled set needs to span the situations your feature actually encounters, not just the easy middle. Define the dimensions that describe real variation and generate or sample across them. For a user-facing assistant, a workable set of dimensions is feature, scenario, and persona: which capability is being exercised, what situation the system is in (multiple matches, no matches, ambiguous request, invalid data, system error, unsupported request), and who is asking [2].

That taxonomy is not universal, and personas are irrelevant if no human is in the loop. The point is to choose dimensions that matter for your product and cover them [2].

Use real traffic where you have it. Where you do not, generate synthetic user inputs (not synthetic responses) and run them through your actual system so the traces are real [2]. Keep generating until you stop seeing new failure modes.

Step 3: Binary judgments with written critiques

Have the expert answer one question per example: did the system achieve the desired outcome, pass or fail. Then have them write a critique explaining the reasoning.

Binary, not a scale. Nobody knows what to do with a 3 versus a 4, adjacent points mean different things to different annotators, detecting differences needs larger samples, and annotators drift to the middle to avoid deciding [2]. Worse, when you analyze expert judgments against multi-dimensional 1-to-5 scores, the scores usually do not correlate with what the expert actually cares about [2]. If someone insists on rating eight dimensions on a five-point scale, that is usually a sign nobody has decided what matters yet.

The critique is the valuable part. It captures the nuance a binary verdict loses, explains why a flawed output still passed or why a plausible one failed, and becomes the raw material for the judge prompt. Write critiques detailed enough that a new employee could follow the reasoning. Terse critiques are the single most common cause of a weak judge [2].

Remove friction from the review. Put everything the expert needs on one screen: the interaction, user metadata, system state, and whatever lets them verify the answer was actually correct. A simple purpose-built review app pays for itself quickly [2].

Step 4: Fix what you find, then build

Reviewing labeled data will surface real bugs. Fix them before building the judge and go back to labeling. There is no point aligning a grader against a system you are about to change [2]. Iterate until the system is stable.

Step 5: Write the judge prompt from the expert’s own critiques

Build the prompt out of what the expert produced: the criteria they articulated, plus few-shot examples pairing real inputs and outputs with their actual critiques and verdicts. Structure each example with clear delimiters so the judge sees input, output, critique, and verdict as distinct fields, and ask the judge to write its critique before giving its verdict rather than after [2].

The recurring mistakes in judge prompts are all about the examples [2]:

• No critiques at all, just labels.
• Critiques too terse to convey the reasoning.
• Examples missing the external context the expert used to decide, such as user metadata or system state.
• Examples too similar to each other, so the judge never generalizes.

On model choice, use the most capable model your cost and latency budget allows, at least while establishing alignment; optimize for cost afterward [2]. Using the same model family as your main system is usually acceptable for a narrow binary classification task, since what matters is measured agreement with human labels rather than theoretical independence, though switching families is a reasonable move if you cannot reach alignment [3].

Step 6: Validate it like a classifier

This is the step that separates a judge you can cite from one you cannot.

Treat the judge as a binary classifier and split your labeled examples three ways [2]:

Dev and test examples must never appear in the prompt. Aim for 30 to 50 examples of each label in both dev and test where you can; below roughly 60 examples total, confidence intervals get too wide to conclude anything useful [2].

Report true positive rate and true negative rate, not agreement. Raw agreement is only meaningful when your classes are balanced. If a failure occurs in 5% of cases, a judge that always says “pass” achieves 95% agreement while catching none of them [2].

A worked example with illustrative numbers. Suppose your test set has 100 examples, 30 of which the expert marked fail:

True positive rate (failures caught) is 24/30, or 80%. True negative rate (passes correctly left alone) is 65/70, or 93%. Raw agreement is 89%, which flatters the judge by hiding that it misses one failure in five. Whether 80% is good enough is a product decision that depends on what a missed failure costs.

How much iteration to expect. In one documented case, three rounds of prompt revision against a domain expert’s labels reached over 90% agreement, though this varies with task complexity [2]. Track your agreement metrics across iterations so you can see whether you are converging or thrashing.

Correct your production estimates for judge error. Once you know the judge’s sensitivity and specificity, you can adjust the raw pass rate it reports on unlabeled traffic rather than taking it at face value. Reporting the raw proportion is statistically unsound, because imperfect sensitivity and specificity bias the estimate; published methods use a human-labeled calibration set to debias the score and build confidence intervals that account for uncertainty in both the test data and the calibration data [4]. At minimum, report your judge’s error rates alongside any number it produces.

Design around the known biases

The research literature has documented judge failure modes specific enough to engineer against.

Position bias. The order in which candidates appear changes verdicts. A systematic study across 15 judges and roughly 150,000 evaluation instances found position bias is not random noise, varies substantially by judge and by task, and grows as the quality gap between candidates narrows [5]. Mitigation: prefer single-output pass/fail judgments over pairwise comparisons; if you must compare, run both orderings and discard inconsistent verdicts.

Verbosity bias. Judges tend to favor longer answers regardless of quality [6]. Mitigation: say explicitly in the rubric that length is not a criterion, and check whether your judge’s verdicts correlate with output length.

Self-preference. Judges show some preference for text from their own model family [6]. Mitigation: it matters less for narrow binary tasks than for open-ended preference ranking, but if alignment stalls, try a different family.

Temperature sensitivity. Repeat-run agreement is high at temperature zero and degrades substantially as temperature rises [7]. Mitigation: run judges at temperature zero, always.

Stability is not correctness. A judge that consistently favors whatever appears first will score perfectly on repeat-run consistency while being maximally biased [7]. Consistency checks are necessary and not sufficient; you still need human labels.

One more caution: human labels are the reference, not perfection. Human raters have documented tendencies of their own, including rating confident, assertive answers above accurate but hedged ones [8]. Write rubrics that tell annotators what to ignore as well as what to look for.

Step 7: Use it, then keep checking it

Once the judge clears your bar, run it on unseen data and segment the results. Failure rates broken down by scenario, feature, user type, document format, or language tell you where to invest. Aggregate pass rates rarely do [2].

Re-validate the judge whenever something material changes: a new model for the main system, a new model for the judge, a significant prompt revision, or a shift in traffic. There is no shortcut here; you run the process again and measure [2].

You do not need the expert to grade everything forever. A representative sample is enough, and the human effort naturally shrinks as the system matures [2]. But it never reaches zero, because the judge has to stay aligned to something, and that something is a person.

What if you cannot get alignment?

If the judge will not converge with the expert, resist the urge to keep tuning the prompt. The usual root causes are worth addressing directly [2]:

• The feature is overscoped. Narrow it until “good” is definable.
• The process was shortcut. No principal expert, no real critiques, examples reused across splits.
• The criterion is genuinely subjective. Some judgments will not automate, and leaning on human review for those is a legitimate answer.

Either way, you end up with something valuable: a measured statement of how much the judge can be trusted, rather than an assumption.

The thing nobody tells you

The judge is often not where the value comes from. The process forces a domain expert to examine outputs systematically and articulate criteria they had never written down, and that alone tends to surface more insight about the product than the resulting grader does. Experts frequently report realizing mid-process that they had been judging edge cases inconsistently [2].

Which is worth knowing if your goal is a better product rather than a better dashboard. If looking at your data is cheap and you are your own domain expert, you may not need a judge at all, at least not yet. What you can never skip is looking at the data [2].

Checklist

• A code assertion was ruled out before committing to a judge.
• One principal domain expert owns the quality standard.
• The labeled set covers the dimensions that actually vary in production.
• Judgments are binary, with written critiques detailed enough to reuse.
• The judge prompt is built from the expert’s real critiques and real context.
• Labels are split into train, dev, and test, with dev and test never in the prompt.
• Performance is reported as true positive and true negative rates, not agreement.
• The judge runs at temperature zero, with position and verbosity bias addressed.
• Production estimates account for the judge’s known error rates.
• Re-validation is triggered by model or prompt changes, not left to memory.

Keep reading

• LLM evals and LLMOps: how to know your AI feature actually works (the pillar for this series)
• Error analysis for LLM failures
• RAG evaluation: retrieval and generation
• Regression testing AI features in CI
• Monitoring LLM quality in production

CMW AI Labs builds and validates judges the way this guide describes, against labels from your domain experts rather than off-the-shelf rubrics. If you have an eval dashboard you do not trust, talk to our team.

Sources

1. Hamel Husain, “The Revenge of the Data Scientist,” March 2026. https://hamel.dev/blog/posts/revenge/
2. Hamel Husain, “Using LLM-as-a-Judge For Evaluation: A Complete Guide,” October 2024, updated September 2026. https://hamel.dev/blog/posts/llm-judge/
3. Hamel Husain and Shreya Shankar, “AI Evals: Everything You Need to Know,” May 2025, updated September 2026. https://hamel.dev/blog/posts/evals-faq/
4. Lee et al., “How to Correctly Report LLM-as-a-Judge Evaluations,” arXiv:2511.21140. https://arxiv.org/abs/2511.21140
5. Shi et al., “Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge,” arXiv:2406.07791. https://arxiv.org/abs/2406.07791
6. “Testing the Testers: Human-Driven Quality Assessment of Voice AI Testing Platforms,” arXiv:2511.04133, 2025. https://arxiv.org/abs/2511.04133
7. “Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models,” arXiv:2606.19544, 2026. https://arxiv.org/abs/2606.19544
8. Galileo, “LLM-as-a-Judge vs Human Evaluation,” February 2026. https://galileo.ai/blog/llm-as-a-judge-vs-human-evaluation