How much does position bias change your verdict?
A click means someone looked at a result and liked it. Attention is decided by whichever ranker produced the log, so counting clicks measures relevance times position - and it rewards a candidate only for agreeing with production. Weighting each click by the inverse chance its position was examined removes that, but the strength of the bias, eta, has to be assumed rather than measured. Drag it.
What the click log says
| Estimate of DCG credit per query | Logged ranker | Candidate |
|---|---|---|
| Counting clicks | ||
| Position-corrected |
Clicks by shown rank
Bars are what happened; ticks are where pure examination at the current eta would put each rank, anchored to rank 1. The gap between them is relevance - which is exactly why you cannot read eta off this curve.
The catch
On human relevance labels the candidate genuinely wins, by +0.036 NDCG@10 (95% CI [+0.014, +0.057], permutation p = 0.0012). The click log only agrees once eta passes 0.94. Since nobody hands you eta on real traffic, a crossover that close to your assumption means the offline answer is not decisive - go and run an interleaving test.
400 queries and 425 clicks from examples/ in the repository,
simulated at a known eta of 1.0 so the arithmetic is checkable. The same
numbers come out of rankkit bias examples/clicks.jsonl
examples/run_b.jsonl.