LLM as a Judge: When to Trust the Verdict

Short answer

• With "clear" twice, my AI judge's verdict held in both orders in 482 of 482 pairs, and in four of five with at least one "slight".
• The Claude text, longer on average, won 70% of Claude Opus 5.5's verdicts and 32.5% of GPT-5.6-Terra's.
• Send slight or important verdicts to a human or another vendor's judge, and accept only clear ones.

If you have a large language model (LLM) grade AI output, you need the dividing line between "clear" and "slight".

In 10 pairs where Claude and Gemini had analysed the same face photos, a Claude judge picked the Claude analysis every time. A judge from OpenAI, with no analysis of its own in the comparison, picked the Gemini one in 7 cases. A sample that small proves nothing, so I measured whether a judge sides with its own model family.

What I measured and how I counted

Part 1: the archive

From 2 to 29 October 2026, Claude Opus 5.5 made 1,687 blind pairwise verdicts for my own projects. It judged email designs, logos, short-video opening frames, 3D figurine stills, a product photo set and blog texts.

Both orders are on record for 843 pairs. One verdict has no counterpart in the other order and counts only towards the position effect. I gave each project its own question. For designs, logos and images, I asked which draft is the stronger design, for short videos which one better keeps a viewer watching, and for blog texts which of two versions of the same article sounds more like a human. The dataset records a criterion for almost all verdicts, and for blog texts that criterion is mostly the language. The logged cost is USD 110.96.

Terms and rules

• Pairwise comparison: the judge sees two anonymous drafts and names a winner.
• Both orders: every pair runs twice, once with A first and once with B.
• Same verdict: a pair counts as stable when the same draft wins both runs.
• "Clear" or "slight": with every verdict, the judge itself says whether the winner is clearly or only slightly ahead. The answer template offers only the words "clear" and "slight" for this, without defining them.
• Position effect: the share of verdicts won by the draft shown first. At 50%, position gives no advantage.
• Wilson interval: a 95% range where the true share plausibly lies, given this sample.
• p-value: how easily chance alone could produce a result this lopsided. Below 0.05, it usually counts as statistically significant. For the position effect, I use a binomial test against 50%.
• Sign test: for each text pair, I compare which of two judges gave the Claude text more wins. Ties drop out, and the test checks whether the rest could be this one-sided by chance.
• Control pair: a text against a copy with three planted errors. Two are statements that contradict the brief (usually a number and a name or place), and one is a broken sentence. The right answer is known, and a control passes only if the clean text wins in both orders.

Before computing the numbers, I wrote down the counting rules for both parts and froze them with a checksum, which makes any later change to them visible. The list above is a summary, and the full wording comes with the dataset.

Before the freeze, I had done a rough count of the archive from part of its logs. In Part 2, before the first judge call, I also added the language to the writing models' instruction (see the side finding). The sign test and the count of how often two judges picked the same text are not part of the frozen rules.

Nine in ten pairs survive the swap

In 776 of 843 pairs, the same draft won in both orders. That is 92.1%, with an interval of 90.0 to 93.7%. The draft shown first won 857 of 1,687 verdicts, or 50.8%, at a p-value of 0.53. Overall, the position gave no measurable advantage.

Zheng et al. (2023) showed AI judges two deliberately similar answers and swapped the order. GPT-4 kept its verdict in 65.0% of cases, against 92.1% in my archive. Claude-v1 picked the answer shown first both times in 75.0%.

My figures compare with research only roughly. The study tests older models, and its pairs are almost equally good. My pairs come from real work, where two drafts often differ clearly, and clear differences make a verdict more stable.

When a verdict holds: "clear" against "slight"

Source: my own measurement, Claude Opus 5.5 as judge, 2 to 29 October 2026.

In my data, the judge's own rating of its margin meant something. When it said "clear" twice, the verdict held in 482 of 482 pairs, which is no guarantee for the next pair. With at least one "slight", it flipped in 67 of 361 pairs.

Zheng et al. measured 98.8% consistent verdicts where the two models differed a lot in quality, and write that the position effect "can almost disappear" in such cases. Shi et al. (2025) analysed over 150,000 verdicts from 15 judges and call the effect "strongly affected by the quality gap".

In a discussion on Hacker News, a forum mostly for developers, several comments doubt that a model's rating of its own confidence means anything. For this one judge and my tasks, it did. A judge can still say "clear" twice and be wrong twice.

Blog texts: the biggest edge for the first text, but not significant

Blog texts were stable in 58 of 78 pairs, a lower share than average. The text shown first won 90 of 156 verdicts, which is 57.7% at a p-value of 0.065. That is not statistically significant.

Zheng et al. saw verdicts on writing tasks hold after the swap in 42% of cases, and the answer shown first win both times in 46%. Verdicts on maths and coding questions held in 86%.

My guess is that two texts differ less clearly for a judge than two logos do.

Part 2: does a judge favour its own model family?

The setup

• Briefs from invented Swiss small firms, 10 in German and 10 in English, each answered once by Claude Opus 5.5 and by OpenAI's GPT-5.6-Terra (GPT-5.6 from here on).
• They ask for everyday texts such as customer emails, website copy, a payment reminder or a rejection to an applicant.
• Opus, the smaller Claude Haiku 5.5 and GPT-5.6 judged every pair in both orders with the same instruction. I asked them which text a demanding editor at a Swiss business publication would send unchanged, weighing the brief and correct facts first, then clarity and tone, not length.
• Plus 6 control pairs with planted errors, 3 per language, half from each writer's texts.
• Each judge rated 26 pairs, controls included, and gave 40 verdicts on the 20 text pairs.

What the research leads you to expect

Panickssery et al. (2024) had GPT-4 compare news summaries from two datasets. On one, GPT-4 chose its own summary in both orders in 87.7% of comparisons and the other summary in 3.4%. On the other, the figures were 59.3% and 18%. Wataoka et al. (2024) also found GPT-4 favouring its own texts and traced this to a model rating familiar-looking texts higher, its own or not. Both studies test older models.

A comment on Hacker News advises using a judge from another vendor, without a measurement.

"they [the models] are best at critiquing the output of another vendors model" (comment on Hacker News)

Result

Source: my own measurement of 29 October 2026, 156 verdicts.

Opus and GPT-5.6 sit 37.5 percentage points apart. Both judges were stable on 13 of the 20 text pairs, and they picked the same text in 7 of them. They picked different texts in the other 6, and in each of those at least one judge had said "slight". In 3 of the 6, Opus said "clear" twice and GPT-5.6 "slight" twice. On the remaining 7 text pairs, at least one judge changed its verdict after the swap.

The sign test counts differently and sets aside the pairs with the same pick as ties. Of the rest, Opus gave the Claude text more wins than GPT-5.6 did in 11 pairs and fewer in 2, at a p-value of 0.0225.

But the Claude texts averaged 147 words against 117 for the GPT texts and were longer in 19 of 20 pairs. Opus picked the longer text in 12 of the 18 text pairs that were stable for it, and GPT-5.6 in 5 of its 15. Origin and length cannot be separated in this run. Haiku, also a Claude model, sat near half at 52.5%, so this run shows a lean towards the Claude text in Opus, not in the whole family. GPT-5.6 leaned about as strongly towards the text from its own vendor, which was mostly the shorter one, and gave it 67.5% of verdicts. No human rated the texts, so I do not say which is better.

Side finding: the language of the answer

With my first instruction, "in the language of the brief", GPT-5.6 answered eight of the ten English briefs in the invented firm's local language (German 5, French 2, Italian 1). Claude wrote all ten in English. Before the first judge call, I changed the instruction and had all 40 texts rewritten. The new version names the language, for example "in English". If you work at a Swiss company, name the output language in the instruction.

Does an AI judge work the same in German?

Hada et al. (2024) compared GPT-4 with over 20,000 human ratings in several languages, German included. The judge leaned towards high scores, and the authors recommend calibrating it against human ratings for each language.

Fu and Liu (2025) had five judges rate the same content in several languages, 25 in total. Across languages, a judge's verdicts on the same content reached an agreement of about 0.3 on average, measured with Fleiss' kappa, where 0 means agreement by chance alone and 1 means full agreement. I found no study of German business texts that compares a judge with human ratings.

In my run, Opus gave the Claude text 75% of its verdicts in German and 65% in English. Haiku gave it 55% and 50%, and GPT-5.6 gave it 30% and 35%. All 13 German pairs were stable for GPT-5.6 (10 text and 3 control pairs), against 8 of 13 in English. With 13 pairs per language, none of these splits by language generalises.

How I use an AI judge

• Compare instead of scoring. I have two drafts compete. A comment on Hacker News says models are better at comparing two items than at scoring one on its own.
• Always both orders. A draft wins only if it is ahead in both, otherwise the pair counts as a tie. Zheng et al. propose this rule too.
• Slight and important verdicts go to a second check. They go to a judge from another vendor or to a human. A verdict is important when a wrong pick would be costly or hard to undo. Important ones go there even when clear, because in 3 pairs GPT-5.6 contradicted an Opus verdict that was "clear" twice. Anthropic's guide to success criteria and evaluations calls it generally best practice to grade with a different model from the one that produced the text. In Verga et al. (2024), a panel of three small judges from three families did better than GPT-4 alone, at a seventh to an eighth of the cost.
• Mix in control pairs. Anthropic advises testing reliability first and only then scaling up. In my run, Haiku missed one of six controls, in one of the two orders. Its own reasoning there identified the clean text as correct, but its verdict went to the other anyway.
• Code before the judge. I never leave to a judge what a simple program can check, such as whether a required detail is in the text.

The judge is one of several checks on my drafts. AI Blog Writing: How I Check Every Claim shows how I check AI texts before they go out.

Where a judge should not decide alone

In hiring decisions, legal or medical questions and security approvals, a wrong pick costs too much. In those cases, a judge can at most do a first sort, and a human decides.

Limits of this measurement, and the dataset

The archive verdicts were made before I wrote down my method standard for my own AI benchmarks. The result for "clear" twice remains the finding of a single Claude judge, which assigns "clear" and "slight" itself. Part 2 tests new, invented texts with judges from two families and with control pairs.

What I measured is stability, not correctness, and neither part has human ratings to compare against. Part 2 is small, with 20 text pairs and 6 control pairs, and its numbers hold for the models of October 2026. Part 2 cost USD 3.47 in billed calls, including the discarded first round of texts, and the GPT-5.6 runs went through a subscription without per-call billing. The exact model ids are in the dataset.

The dataset is on the standard page for download. It contains archive-calls.csv, rerun-calls.csv, rerun-texts.json and schema.json, and the page describes all four. Next to them are the rules in archive-rules.json and rerun-factors.json, the instruction to the Part 2 judges in rerun-judge-prompt.txt, and the checksums with the times the rules were frozen in SHA256SUMS.txt. One logo pair from the archive looks like this, with anonymous ids (hashes) instead of drafts.

call_id,source_set,modality,criterion,pair_id,entry_a_id,entry_b_id,shown_first,winner,winner_position,margin,judge_model,cost_usd,date
cd99b1636,s324fc5,logo,logo,pe8510359,e16b2fca4,e129ab7bf,e129ab7bf,e129ab7bf,1,clear,claude-opus-5-5,0.05191,2026-10-10
ca3ddbda1,s324fc5,logo,logo,pe8510359,e16b2fca4,e129ab7bf,e16b2fca4,e129ab7bf,2,clear,claude-opus-5-5,0.0458,2026-10-10

I explain the reason for the machine-readable format in Why Most LLM Benchmark Sites Are Invisible to AI Search. Model leaderboards are on my overview of AI benchmarks.

Test a judge before it decides

If you want AI output graded automatically in your SME or team, test the judge on your own data first. Run every pair in both orders and mix in control pairs, where you know which text should win. You can reach me through the contact page to talk it through.

Frequently asked questions