The labels under the benchmark
211,225 crowd ratings sit under GoEmotions. On 27 of its 28 emotions, the raw annotation layer falls below the reliability floor the field normally requires.
Every benchmark score, every fine-tune, every “our judge agrees with humans 92% of the time” bottoms out in a pile of human labels. Those labels were produced by people who may have been careless, biased, tired by the end of the queue, or graded against gold that was quietly wrong. None of that is visible in the accuracy numbers computed on top — by the time you are reporting F1, the annotation layer has been flattened into ground truth and stops being questioned.
So I went looking underneath one of the well-known ones.
211,225 ratings
GoEmotions is a widely used emotion-classification dataset: 58,011 Reddit comments, 28 binary emotion labels each, 211,225 individual ratings from 82 raters. The released splits aggregate and filter those raw ratings, which is standard and sensible. The raw layer is published too, and that is what I audited.
What the ratings say
27 of the 28 emotions sit below the 0.667 reliability floor — and that is the whole 95% confidence interval, not just the point estimate. Only gratitude clears it, at alpha 0.72. The median emotion manages 0.23. Neutral, the single largest and apparently most straightforward category, scores 0.26.
Rater quality spans a factor of three. Measured as each rater against the rest, across the 67 raters with 300 or more ratings, agreement runs from 0.11 to 0.36. Twenty of those 67 sit below 0.20. The noisiest raters agree with their co-raters at about a third of the rate the best ones do, and every one of them contributed to the pool equally.
The rater count was never enough for the task. This is the finding I find hardest to argue with, because it does not depend on any judgement about who was careless. Given the observed per-rater reliability, Spearman–Brown says you would need roughly 13 raters per item to produce aggregated labels that are 0.8-reliable. The dataset averages 3.6, which projects to 0.52. The aggregation is doing what aggregation does; there is simply not enough of it.
Disagreement without awareness of disagreement. 86% of multi-rated comments are contested on at least one emotion. Raters marked just 1.6% of their ratings “very unclear.” People were confident and divided at the same time, which is the signature of a task where the label boundary is genuinely underdetermined rather than merely difficult.
What this is not
None of this says GoEmotions is a scandal. Fine-grained emotion is genuinely hard — ask two people whether a comment expresses annoyance or disapproval and you should expect trouble. The released splits aggregate and filter these raw ratings precisely because the raw layer is noisy, and the dataset has been useful to a great many people, including me.
What it says is narrower and, I think, more useful: the raw layer under a famous benchmark fails the standard reliability bar almost everywhere, measurably and reproducibly. If your model scores 0.61 on an emotion whose annotation reliability is 0.23, the interesting question is no longer whether 0.61 beats 0.59. There is a ceiling in the data, and it is lower than the numbers being compared.
The general point
I do not think GoEmotions is unusual. I think it is unusually well documented — the raw ratings are public, which is exactly why this audit is possible at all. Most datasets do not release the annotation layer, and so the question cannot be asked.
The checks are not expensive: reliability per label with an interval, per-rater agreement against the rest, drift across the annotation queue, duplicate items across the train and test split, and how often the gold itself disagrees with a competent majority. Running them takes minutes. Not running them means the first time anyone learns the labels were shaky is when a model refuses to improve and nobody can say why.