What an LLM judge actually does when you are not looking
Three local models were asked to grade the same arithmetic, with one answer provably correct. One of them picked whichever answer was shown first 91% of the time.
LLM-as-judge is now the default way to evaluate models, agents and prompts. It is cheap, it scales, and it correlates with human ratings well enough to pass review. The problem is the thing everyone knows and few people measure: a judge is a model too. It can prefer whichever answer came first, reward padding, favour its own outputs, hug the middle of the scale, and say “90% sure” while being right 70% of the time — all while agreeing with your annotators often enough to look fine.
So I built a benchmark where the right answer is not a matter of opinion, and pointed three judges at it.
The setup
The task is adversarial arithmetic. Each item is a pair of candidate answers to a small arithmetic problem, one of which is provably correct by construction — there is no rubric to argue about, no style to reward, no ambiguity for a judge to hide behind. Half the time the wrong answer is the long, worked-through one, full of plausible intermediate steps that happen to end somewhere false.
Every pair is then judged twice, once in each presentation order. That second pass is what makes the position effect measurable rather than inferable: if a judge’s verdict changes when the only thing that changed is which answer came first, slot order decided it, not content.
Three judges, three different ways to fail
| Judge (temperature 0) | Truth agreement | Position preference | Verbosity excess | Flips under swap | Decisive on identical pairs |
|---|---|---|---|---|---|
| aya-expanse:8b | 0.433 | 0.910 | +0.106 | 0.819 | 1.000 |
| qwen2.5:14b | 0.667 | 0.768 | +0.131 | 0.568 | 0.200 |
| llama3.1:8b | 0.443 | 0.576 | +0.483 | 0.171 | 1.000 |
Same task, same temperature, three different failure modes at three different severities.
Two of the judges follow slot order. aya-expanse:8b picks whatever is presented first 91.0% of the time, and flips its verdict on 81.9% of pairs when the order is swapped — which is close to saying the model is reading the position and not the arithmetic. qwen2.5:14b is better but not well: 76.8% and 56.8%.
The third fails somewhere else entirely. llama3.1:8b is comparatively steady under swapping, at 17.1%, because it is not grading position — it is grading effort. It picks the longer answer 48 points more often than correctness warrants, and agrees with ground truth less often than a coin flip. A judge that rewards visible work will reward a confident wrong derivation over a terse right answer, every time.
The last column is the one I did not expect. Two of the three judges never once declared a tie between two identical answers. Asked to compare a string with itself, they picked a winner.
The usual caveats apply and they matter: n = 105 pairs, adversarial by construction, three small local models at one temperature. This is an illustration of what the probes can see, not a model leaderboard.
Agreement is not an audit
The obvious objection is that nobody ships a judge without checking it against human labels first. That check is weaker than it looks. Among the synthetic judges I use to validate the probes themselves, there is one whose only implanted defect is that it adds standardised answer length to perceived quality. It reaches a quadratic-weighted kappa of 0.80 against its human reference — a number most eval writeups would celebrate — while carrying a verbosity bias of +0.82.
Agreement is an average. A bias is a direction. An average can look healthy while a direction quietly decides every close call, and close calls are exactly the comparisons an eval exists to make.
What to measure instead
The failure modes above are each individually measurable, and none of them requires gold labels beyond the ones you already have:
- Position preference — judge every pair in both orders and count how often the verdict survives.
- Verbosity excess — how much perceived quality tracks length, over and above how much correctness does.
- Self-preference — whether the judge scores its own family higher.
- Central tendency — whether the scale is being used or hugged.
- Calibration — whether a claimed 90% is right nine times in ten.
Each of these deserves a confidence interval rather than a point estimate, and a flag only when the interval leaves the range an innocent judge could plausibly occupy. Otherwise you have swapped an unexamined judge for an unexamined dashboard.
The part I keep coming back to is that none of this is exotic. The probes are cheap, the swap costs one extra inference per pair, and the answer changes what you are willing to conclude from an eval. It is the same discipline we would apply to a human annotator pool without thinking twice about it.