Aggregate benchmark gains can mask item-level LLM regressions

A study examined three successive commercial API upgrades in what it describes as the GPT-5.4 to GPT-5.6 Sol product sequence, testing 900 public benchmark items covering graduate-level knowledge, olympiad mathematics, and instruction following. Each item was queried 50 times per model, and results were classified as reliably improved, reliably regressed, practically equivalent, or inconclusive using false-discovery-rate control and a permutation-based null baseline. Across all nine migration-benchmark combinations, reliable improvements and reliable regressions coexisted: edges with aggregate gains up to 7.3 percentage points still contained up to 8.3% reliably regressed items, and edges with aggregate losses contained up to 10.7% reliably improved items. On the instruction-following benchmark, a 3.9-point regression under strict scoring shrank to 0.04 points under loose scoring, showing that scoring method choice can hide the same underlying change. The full response archive and per-item scoring outputs were released alongside the paper.

rss · arXiv cs.SE · Aug 19, 04:00

「Why aggregate benchmarks were trusted for migration decisions」 Organisations that build production systems on commercial LLM APIs are periodically forced to migrate when vendors deprecate older model versions, as is occurring across the GPT-5.4 to GPT-5.6 Sol sequence referenced in this study. The standard practice for approving such a migration is to compare aggregate benchmark scores between the old and new model versions and treat a net positive delta as evidence that the upgrade is safe to deploy. This assumption rests on the idea that a single compressed score adequately represents how a model performs across the many distinct items and tasks a benchmark contains, rather than obscuring gains and losses that offset one another.

「Who this affects」 This concerns any organisation that migrates production systems to a successor commercial LLM API version based on vendor-published or self-run aggregate benchmark deltas, without item-level or task-level regression testing. Teams relying on instruction-following behaviour are particularly exposed, since the study found scoring strictness alone can flip a measurable regression into an apparent non-event. The measured findings come from a specific vendor sequence and three pairwise upgrades, so organisations using other model families or providers should treat this as a demonstrated failure mode to check for, not a rate that transfers directly.

「Mitigation」 There is no vendor-side fix implied here; the finding argues for organisations to adopt item-level or task-level regression testing with repeated sampling and statistical significance controls before migrating production traffic to a new model version, rather than relying on a single aggregate score comparison. The released response archive and per-item scoring outputs can serve as a reference for building such tests.

Tags: #LLM evaluation, #model migration, #benchmark validity, #regression testing, #vendor claims