Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Benchmark^2: Systematic Evaluation of LLM Benchmarks

Qi Qian, Chengsong Huang, Jingwen Xu, Changze Lv, Muling Wu, Wenhao Liu, Xiaohua Wang, Zhenghua Wang, Zisu Huang, Muzhao Tian, Jianhan Xu, Kun Hu, He-Da Wang, Yao Hu, Xuanjing Huang, Xiaoqing Zheng

TL;DR

Benchmark^2 tackles the problem of evaluating the quality of LLM benchmarks themselves. It introduces three metrics—Cross-Benchmark Ranking Consistency, Discriminability Score, and Capability Alignment Deviation—and an aggregate Benchmark Quality Score to quantify benchmark reliability, discriminability, and hierarchical alignment. Through a large-scale analysis of 15 benchmarks and 11 models across four families, the study shows substantial benchmark quality variation and demonstrates that high-quality subsets can match full-benchmark evaluation with about 35% of the data. The framework is validated with held-out models and ablation analyses, and it offers practical guidance for constructing efficient, reliable benchmarks. Collectively, Benchmark^2 provides a rigorous, scalable approach to improve the integrity of LLM evaluation.

Abstract

The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets.

Benchmark^2: Systematic Evaluation of LLM Benchmarks

TL;DR

Benchmark^2 tackles the problem of evaluating the quality of LLM benchmarks themselves. It introduces three metrics—Cross-Benchmark Ranking Consistency, Discriminability Score, and Capability Alignment Deviation—and an aggregate Benchmark Quality Score to quantify benchmark reliability, discriminability, and hierarchical alignment. Through a large-scale analysis of 15 benchmarks and 11 models across four families, the study shows substantial benchmark quality variation and demonstrates that high-quality subsets can match full-benchmark evaluation with about 35% of the data. The framework is validated with held-out models and ablation analyses, and it offers practical guidance for constructing efficient, reliable benchmarks. Collectively, Benchmark^2 provides a rigorous, scalable approach to improve the integrity of LLM evaluation.

Abstract

The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets.
Paper Structure (57 sections, 10 equations, 2 figures, 17 tables)

This paper contains 57 sections, 10 equations, 2 figures, 17 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Benchmarks
  4. Benchmark Quality Analysis
  5. Methodology
  6. Cross-Benchmark Ranking Consistency
  7. Definition.
  8. Interpretation.
  9. Discriminability Score
  10. Definition.
  11. Interpretation.
  12. Capability Alignment Deviation
  13. Model Family Hierarchy.
  14. Definition.
  15. Interpretation.
  16. ...and 42 more sections

Figures (2)

  • Figure 1: Overview of Benchmark$^2$ framework. Top row: Three key problems with existing LLM benchmarks—ranking inconsistency across benchmarks, low discriminative power of performance gaps, and prevalence of rank-inconsistent test items. Bottom row: Our three complementary metrics addressing each problem—Cross-Benchmark Ranking Consistency (CBRC) measures alignment with peer benchmarks, Discriminability Score (DS) quantifies performance gap magnitudes, and Capability Alignment Deviation (CAD) identifies items violating expected capability hierarchies within model families.
  • Figure 2: Effect of selection ratio on benchmark quality metrics. The optimal point at 35% (marked) achieves a good balance between ranking consistency ($\tau$ = 0.93), stability (0.69), and discriminability (DS = 0.47).