Open-source libraries for measuring whether a judge, an experiment, a leaderboard or a dataset can be trusted. Installable, tested, and archived with a DOI.
My doctoral work asks when a language model’s judgment can be trusted: of text, of annotations, of another model. These libraries are that question turned into tools.
Each does the same job for a different artefact. It reads what you already have (a file of verdicts, an experiment readout, a pile of ratings, a board’s votes), measures the specific ways that artefact can mislead you, puts a confidence interval on every number, and raises a flag only when the evidence clears a stated bar.
They are validated the same way, too. Every probe is checked against synthetic cases with a defect implanted on purpose: it has to fire on the case carrying the defect and stay silent on the clean one. That table sits at the top of each README, regenerated by CI, so it cannot drift from what the package does.
LLM-as-judge
Which way the judge leans, and by how much.
Position preference, verbosity bias, self-preference, central tendency and over-confidence, each with a bootstrap interval. One synthetic judge in its own test suite agrees with humans at 0.80 quadratic-weighted kappa while carrying a verbosity bias of +0.82. Agreement is not an audit.
Estimate judge accuracy when there are no gold labels.
When several imperfect judges label the same items, the pattern of their disagreement reveals who is reliable. A Dawid–Skene fit recovers each judge’s confusion matrix, the class prevalence, and a posterior over every item’s true label, weighted by judge quality rather than one judge, one vote. Pure NumPy, deterministic.
What outcome-only evaluation misses about agents.
If the final answer is right, outcome-only judging cannot see that the agent skipped a required check, acted against what a tool returned, or promised something no observation supports. A synthetic environment with those faults injected on purpose measures the gap between outcome-only, step-rubric and self-consistency judging.
How much of the order survives its own uncertainty.
arenakit fits the same Bradley–Terry model the public boards use, then reports which parts of the published order the evidence actually carries: rank volatility, position bias, intransitive cycles, halves of a board that never met, and comparison budgets spent mostly at the top.
Applied statistics
Sample-ratio mismatch, peeking, multiple testing and the winner's curse.
A readout can say “B wins, p = 0.03, ship it” while any of those is true, and none of them shows up in the p-value. abkit measures each failure mode from the per-unit data and flags what leaves the range a healthy experiment could occupy.
Treat an eval like the A/B test it is.
Eval scores are sample estimates reported as exact numbers: “our prompt scores 76.5%, the old one 63.5%, ship it.” On two hundred items, much smaller gaps than that are noise. Confidence intervals for one run, paired tests for two on the same items, sample-size planning before you spend on inference. Standard library only.
pip install abeval
A model that says 90% should be right nine times out of ten.
Most are not: modern classifiers, and LLM judges scoring on a 1–10 scale, are routinely over-confident, and accuracy alone never shows it. Reliability diagrams, expected calibration error and Brier decomposition with intervals, a significance test that separates real miscalibration from sampling noise, and the temperature that fixes it. No dependencies.
Data quality
What the raters agreed on, and where they did not.
Raters get careless, biased, tired by the end of the queue, or graded against gold that was quietly wrong. raterkit measures reliability, rogue and leaning raters, competence drift, split leakage and stale gold. On the raw GoEmotions ratings, 27 of 28 emotions fall below the 0.667 agreement floor.
Ranking metrics with error bars, and click metrics that survive position bias.
“NDCG@10 went from 0.826 to 0.862, ship it” is missing two things. An interval: on a few hundred queries, a gap that size is routinely noise. And a correction, because if your labels are clicks rather than judgements, the metric is measuring what the incumbent ranker put on top, and will quietly tell you every candidate is worse than production.
Explain the classifier, then test the explanation.
Most token heatmaps are never tested. explainkit computes word-level attributions for anything behind a predict_proba-style callable, then makes them earn their keep: select the tokens carrying 95% of corpus importance, steer an annotator toward them, swap them out to see whether the verdict survives, and score the chain for faithfulness, plausibility and cross-method agreement. The pipeline behind the published work on transparent text classification.
Judges drift when providers update models behind stable API names. judgewatch runs a frozen probe set of 36 hand-written items against popular judge models every month and publishes the result, so a judge that changed under you is visible. It measures position flips, verbosity preference, susceptibility to a fabricated “9 out of 10 experts prefer…” line, and self-consistency. The first run is pending; the method and the probe set are public now.
Counterfactual learning-to-rank for marketplace search logs, end to end in PySpark: position-bias estimation from randomized traffic only, inverse-propensity-weighted training, and NDCG against known ground truth. Because the log is synthetic with a known generative process, correctness is provable in tests: the propensity estimator has to recover the generator’s position-bias curve, and the re-ranker has to beat the logged production ordering. No UDFs anywhere.
Machine learning fundamentals written from scratch in numpy. In most “from scratch” repositories nothing is checked against anything, and the claims in the prose (regularisation helps, bagging reduces variance) are assertions rather than measurements. Here every estimator is pinned by a test against an independent reference, every table is regenerated from the code, and CI fails the build if a committed number no longer follows from committed code.
Replication code for individual papers sits with the paper. EvalMORAAL is a packaged evaluation framework for moral alignment in LLMs, released with its *SEM 2026 paper; the sexism-detection, token-replacement and cultural-alignment studies each ship their own repository from the publications page. Applied and earlier engineering work (dashboards, optimisation, simulation) is on the projects page. The thirteen course notebooks for Opening the Black Box are open too.