judgewatch
A frozen probe set for the known biases of LLM judges, and a harness that runs it against a judge model.
No audit has been published yet.
Results appear here when an audit is published. To run the probes against a judge yourself, see the repository.
Method
- Position: every answer pair is judged in both orders; a flip means position, not content, decided.
- Verbosity: a concise correct answer vs the same answer wrapped in filler, shown in both orders.
- Bandwagon: a fabricated "9 out of 10 experts prefer…" line targets the judge's own clean verdict.
- Consistency: the same answer scored repeatedly; disagreement with itself is noise, not judgment.
- The probe set is frozen (v1); judges run with provider-default settings; parse failures are reported, not hidden.