What a benchmark can actually tell apart

A leaderboard row says 78.2 against 75.4 and everyone reads it as a result. Whether it is one depends on something the row never shows you: how many items the benchmark has. If the row is an average over ten benchmarks it has more items behind it than any single column, so what follows is about the columns.

Every eval score is an estimate from a sample. Run two systems on the same items and the gap between them is part real difference and part luck of the draw. The item count puts a floor under how small a real difference you can reliably notice through that noise. By the floor I mean the smallest true gap the benchmark catches four times in five; below it, a real difference is more likely to be missed than found.

So I went and worked out the floor for five benchmarks people quote every week.

The floors

Benchmark	Items	Smallest gap it can resolve
HumanEval	164	9.3 points
GPQA Diamond	198	8.4 points
SWE-bench Verified	500	5.3 points
GSM8K (test)	1,319	3.3 points
MMLU (test)	14,042	1.0 points

Only MMLU gets near a one-point floor, and it lands almost exactly on it. For the other four, a true three-point gap, which most of us would read off a table and treat as settled, is one they catch well under half the time.

HumanEval is the one that stops me. It has 164 problems. A twenty-point gap on it was always real, and the early comparisons had gaps like that. The floor only bites the close calls, and the calls got closer as the benchmark saturated. For two models a true three points apart today, the ordering still comes out right about four times in five, but the benchmark can certify it about one time in seven. You will usually see the right winner and almost never be able to prove it.

What is behind the numbers

These come out of a paired power calculation: 80% power, 95% confidence, both systems scored on the same items, a baseline around 75%, and an item-level correlation of 0.5 between the two runs. One thing that does not help: sampling many completions per problem. That averages out decoding noise, but 164 problems are still 164 draws from the space of problems, and the floor counts problems. Every assumption matters, so here is what happens when you move them.

A higher baseline shrinks the floor, because scores near the ceiling vary less. At 90% instead of 75% the floors become 6.1 for HumanEval, 5.6 for GPQA Diamond, 3.5 for SWE-bench Verified, 2.2 for GSM8K and 0.7 for MMLU. Three of the five are still above three points.

Correlation is the other lever, and it is the reason paired designs are worth the trouble. When two systems succeed and fail on the same items, and they usually do, the shared difficulty cancels out and you need far fewer items than two independent runs would need. I used 0.5, a middling assumption for two variants of one system. More alike than that and your floor is lower; two genuinely different models can sit nearer 0.3, where HumanEval's floor grows to 11 points.

One thing this is not: a claim that any particular published gap is wrong. A floor tells you what the benchmark could have detected, not what happened. A gap under the floor might still be real. You just cannot tell from that benchmark, which is a different and more awkward position than being wrong.

Doing the arithmetic first

The useful version of this is not auditing other people's numbers. It is running the calculation before you spend on inference, which takes about a minute.

pip install abeval

Ask how many items you need before you build the set:

abeval power --baseline 0.75 --delta 0.03

That prints the item count for the difference you care about, plus what you could detect with 100, 200, 500 and 1,000 items. For a three-point difference at a 75% baseline the answer is 1,568 paired items, which is more than most eval sets anyone builds in an afternoon.

Then, once you have two runs scored on the same items:

abeval compare run_a.jsonl run_b.jsonl

It gives the difference with a confidence interval and a sign-flip permutation test, which needs no normality assumption, and it drops any item that is not in both files rather than quietly comparing different sets.

Small is not the same as bad

None of this makes small benchmarks worthless. GPQA Diamond is small because graduate-level questions written and validated by domain experts are expensive, and that expense is the whole point of it. HumanEval has been useful to a very large number of people at 164 problems.

The problem is not the size. It is reading a two-point gap off a 198-item benchmark as though it settled an argument, when the same benchmark would show a two-point gap between a system and itself often enough to be unremarkable.

The approach here follows Miller’s Adding Error Bars to Evals, which makes the case at length: report standard errors, use paired designs, plan the sample size. abeval is that advice with a command line attached, so the arithmetic happens before the claim rather than after somebody questions it.