A Rosetta Stone for AI Benchmarks

TL;DR

This work tackles the problem of rapidly saturating AI benchmarks by introducing a simple statistical framework that stitches disparate benchmarks into a single quantitative scale of model capability and benchmark difficulty, without assuming explicit compute-time relations. The approach yields time-series of capabilities, enables forecasting of future capabilities, and permits estimation of algorithmic progress through training compute, while also enabling detection of rapid accelerations. Validation with Epoch AI benchmarking data shows that frontier-model progress is detectable and broadly aligns with prior progress literature, though interpretation remains nuanced due to benchmark biases and the single-number abstraction. The paper further demonstrates synthetic-data and historical-data methods for acceleration detection, highlighting potential for early warning signals of rapid AI progress while acknowledging substantial uncertainty and calls for richer, item-level data in future work.

Abstract

Most AI benchmarks saturate within years or even months after they are introduced, making it hard to study long-run trends in AI capabilities. To address this challenge, we build a statistical framework that stitches benchmarks together, putting model capabilities and benchmark difficulties on a single numerical scale. This acts as a "Rosetta Stone", allowing us to compare models across a wide range of abilities and time, even if they are not evaluated on the same benchmarks. Moreover, this works without assuming how capabilities evolve across time or with training compute. We demonstrate three applications of this framework. First, we use it to measure the speed of AI progress over time, and to forecast future AI capabilities. Second, we estimate the rate of improvements in algorithmic efficiency, finding estimates that are higher, but broadly consistent with prior work. Finally, we find that our approach can be used to detect rapid accelerations in AI progress.

Figures (21)

• Figure 1: Estimated model capabilities $C_m$ and benchmark difficulties $D_b$ over time. 0 corresponds to the difficulty of the WinoGrande benchmark. We determine error bars through sensitivity analysis. Specifically, we perturb one model's capabilities and calculate the $L^2$ distance between the perturbed and original capability/difficulty sets. The error bars indicate the perturbation magnitude required to increase the loss by 5%. We do the same with benchmark difficulties.
• Figure 2: When looking at the most capable models and most challenging benchmarks, our framework's predictions broadly match our intuitions from using models in practice. For example, the state-of-the-art model at the time of writing is GPT-5 or GPT-5.1.
• Figure 3: Our statistical framework and benchmark data predicts model time horizons from kwa2025measuringaiabilitycomplete fairly well, with an $R^2$ of 0.85. The resulting transformation is $\text{time horizon} = \exp(3.69 \times C_m - 4.58)$.
• Figure 4: Predicting benchmark scores using our framework works well on most benchmarks, but there are some exceptions, two of which we show here. For example, Claude models perform better than predicted on SWE-Bench verified, a coding benchmark (left), but worse than predicted on a multimodal benchmark, GeoBench (right). This suggests some degree of specialization, where models from different labs are optimized for different objectives.
• Figure 5: A naive forecast of future frontier model capabilities suggests a capabilities increase of 1.8 additional units over three years, double the improvement between GPT-4o and GPT-5 (high). Here, we forecast future frontier capabilities by extrapolating the capability scores of models that were state-of-the-art at release.