How Reliable is Language Model Micro-Benchmarking?

TL;DR

The paper tackles the reliability of language-model micro-benchmarks, asking whether small subsets can faithfully reflect full-benchmark judgments. It introduces MDAD, a meta-evaluation that quantifies the minimum full-benchmark performance difference required for a micro-benchmark to preserve pairwise model rankings with high probability. Through extensive experiments across MMLU, MMLU-Pro, BIG-Bench Hard, and GPQA, the authors show that very small micro-benchmarks have limited discriminative power for close pairs, while random sampling becomes competitive when micro-benchmarks include hundreds of examples. The findings offer actionable guidance for practitioners on balancing evaluation efficiency and reliability, highlighting that larger micro-benchmarks are often necessary to distinguish similarly performing models, whereas smaller sets can suffice for broad ranking tasks. The MDAD framework provides a nuanced view beyond aggregate rank correlations, enabling more informed micro-benchmark design and interpretation.

Abstract

Micro-benchmarking offers a solution to the often prohibitive time and cost of language model development: evaluate on a very small subset of existing benchmarks. Can these micro-benchmarks, however, rank models as consistently as the full benchmarks they replace? And can they rank models more consistently than selecting a random subset of data points? In many scenarios, we find that the answer is no. We introduce a meta-evaluation measure for micro-benchmarking which investigates how well a micro-benchmark can rank two models as a function of their performance difference on the full benchmark. This approach can determine which model pairs can be ranked correctly by a micro-benchmark, allowing for a finer-grained analysis of the trade-off between micro-benchmark size and reliability. Prior work has suggested selecting as few as 10 examples; we find that no micro-benchmarking method can consistently rank model pairs 3.5 points of accuracy apart on MMLU-Pro or 4 points apart on BIG-bench Hard. In order to consistently rank model pairs with relatively similar performances, we show that often as many as 250 examples must be selected, at which point random sampling is competitive with existing micro-benchmarking methods. When comparing only 8B instruction-tuned models on MMLU-Pro micro-benchmarks with 25 examples, we find that more than half of pairwise comparisons are not likely to be preserved. Our work provides actionable guidance for both micro-benchmark users and developers in navigating the trade-off between evaluation efficiency and reliability.

Figures (17)

• Figure 1: Existing meta-evaluation metrics (e.g. Kendall's tau rank correlation) summarize micro-benchmark performance for MMLU-Pro in the aggregate (top). At extreme dataset reductions, micro-benchmarks can yield high aggregate rank correlation with full benchmarks (top left), but no micro-benchmark has a high probability of agreeing with the full benchmark when ranking model pairs that differ by fewer than 4 points of accuracy (bottom left, gray background). Once enough examples are selected to distinguish such model pairs (bottom right, gray background), random sampling is competitive. See §\ref{['sec:meta-evaluation']} for details and \ref{['fig:seen-results-panel']} for comparisons to another existing measure.
• Figure 2: Agreement and MDAD measures on MMLU-Pro for uniform random sampling and Anchor Points with 300 source models. The three left panels show the probability that a pairwise ranking of models on a micro-benchmark agrees with the full benchmark's ranking, as a function of the accuracy difference between those models on the full benchmark. The rightmost panel summarizes all these agreement curves by showing the minimum detectable accuracy difference between models at each micro-benchmark size, i.e. the accuracy difference at which each curve in the first three panels crosses the 0.8 probability of agreement threshold. Points A-F show how each agreement curve is summarized by MDAD: each point marks where an agreement curve surpasses 0.8 probability. For MDAD, lower values are better. Error bars show 95% bootstrap confidence intervals over 50 trials.
• Figure 3: Comparing six micro-benchmarking approaches on two benchmarks. $y$-axis shows agreement (Equation \ref{['eqn:agreement']}), the probability that a micro-benchmark agrees with the full benchmark when comparing two models, as a function of how much those models differ on the full benchmark ($x$-axis). The rightmost column summarizes agreement curves using MDAD (Equation \ref{['eqn:mdad']}). At small micro-benchmark sizes, all methods struggle to compare models that differ by fewer than 4 points of accuracy on the full benchmark. Anchor Points does best, followed by tinyBenchmarks. Error bars show 95% bootstrap confidence intervals over 50 trials. Figure \ref{['fig:seen-correctness-results-panel-full']} (Appendix \ref{['app:full-results-entire-benchmarks']}) shows all benchmarks.
• Figure 4: MDAD gives more granular information than mean estimation error and Kendall's tau rank correlation. Anchor Points is the only method that consistently outperforms random sampling at small dataset sizes across all metrics. Top row: Mean estimation error. Middle row: Kendall's tau rank correlation. Bottom row: Minimum Detectable Accuracy Difference (MDAD, ours, Equation \ref{['eqn:mdad']}). MDAD panels are the same as in Figure \ref{['fig:seen-correctness-results-panel']}, shown here for ease of comparison. Points A-H labeled for ease of reference in §\ref{['sec:mdad-differences']}. Error bars represent 95% bootstrap confidence intervals over 50 trials.
• Figure 5: When comparing 8B-parameter instruction-tuned models on MMLU-Pro: model accuracies are in a narrow range, so nearly half of pairwise accuracy differences are less than 5 points (left), which is less than the MDAD for micro-benchmarks at small dataset sizes (right).