4,873 experiments and the winner’s curse
Upworthy ran thousands of headline A/B tests and published the raw data. Nearly one in six fails a basic sanity check on its own traffic split.
An experiment readout can say “B wins, p = 0.03, ship it” while the traffic split was broken, three whale customers carried the whole effect, the analyst stopped at the first significant refresh, the winning metric was one of forty tried, or the experiment never had the power to estimate what it now claims. None of that shows up in the p-value.
Those are easy failure modes to demonstrate on synthetic data, where I can implant a defect and check that the probe finds it. The harder question is how often they happen in the wild. The Upworthy Research Archive makes that answerable: 4,873 real headline A/B tests run between 2013 and 2015, with the per-variant data published.
Nearly one in six fails on its traffic split
15% of the tests fail the sample-ratio check at p < 0.001, against the platform’s documented equal-allocation design. A sample-ratio mismatch means the traffic did not arrive in the proportions the experiment intended, and once that is true you no longer know what the two groups differ by. The effect estimate is measuring the assignment as much as the treatment.
The obvious explanation is packages added mid-test, which would make the imbalance a bookkeeping artefact and nothing worse. It does not hold: the rate is, if anything, higher among tests whose variants all launched together. Those readouts carried an exposure bias nobody could have quantified from the summary table.
Most significant winners do not survive being counted
10.9% of the tests have a “significant winner”: the best headline beats the runner-up at raw p < 0.05. Apply Benjamini–Hochberg across all 4,873 winner comparisons, and 2.7% remain.
This is the part that generalises furthest beyond headline testing, because almost nobody runs one experiment. Teams run a programme of them, and each readout is judged on its own p-value as though it were the only test in the world. Across a portfolio, three quarters of the raw winners in this archive do not survive being counted alongside the other 4,872 comparisons, and under a pure null you would still expect roughly 244 of them, about half, to clear p < 0.05 by luck alone.
Even the real winners are overstated
The median significant winning lift is 0.85 CTR points, at a power of 0.62. At that power, the estimate is expected to overstate its true effect by 1.27×, even when the winner really is better.
This is the winner’s curse, and it is not a bias in anyone’s analysis; it is a selection effect baked into the act of choosing. An underpowered experiment only reaches significance when noise happens to push in the same direction as the effect, so conditioning on “it was significant” conditions on a favourable draw. The winner is real and the number attached to it is too big, and the forecast built on that number will miss.
Which is worth sitting with, because a programme that ships only significant winners and forecasts from their measured lifts will systematically under-deliver against its own roadmap, while every individual decision in it was defensible.
What this measures and what it does not
The archive is headline testing on one platform a decade ago, and its click-through rates are far from what most product experiments look like. The sample-ratio analysis treats the archive’s documented equal allocation as the intended design; if some tests intended otherwise and did not say so, those are false alarms. I would not port the exact percentages anywhere.
The structure, though, ports fine. These failures are ordinary. An experimentation programme looks like this whenever each readout is examined on its own. The checks that catch them are cheap, they run on data every team already has, and they are the difference between a decision and a decision you can defend six months later.