A benchmark we trusted turned out to be broken

The field is converging on meta-benchmarking and exerting greater control of how we challenge and evaluate our agents. A growing body of work uses item response theory, the latent-ability model behind standardized testing, to treat difficulty as a measured quantity rather than a label: tinyBenchmarks and metabench show that a handful of informative items reproduce a model's full score, and PSN-IRT turns the same lens on benchmark quality itself. Others audit the ground truth directly: MMLU-Redux found that 6. 49% of Massive Multitask Language Understanding (MMLU) questions are mislabeled, and Platinum Benchmarks re-cleaned ten datasets to minimize both label errors and ambiguity — the same two axes we sweep for. And ambiguity is increasingly treated as intrinsic rather than noise: AmbigQA showed that a large fraction of real questions admit multiple readings, and later work finds that apparent hallucinations often stem from query ambiguity rather than model failure.

What we have not seen elsewhere is the combination: information-theoretic ambiguity sweeping applied as a meta-benchmark over live enterprise data.