Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Aligning Language Model Benchmarks with Pairwise Preferences

Marco Gutierrez, Xinyi Leng, Hannah Cyberey, Jonathan Richard Schwarz, Ahmed Alaa, Thomas Hartvigsen

TL;DR

The paper addresses the gap between benchmark performance and real-world utility by introducing benchmark alignment and BenchAlign, a method that learns preference-aligned weights for test items using a pairwise ranking loss to match a target human-preference ranking. BenchAlign builds a reweighted benchmark that, when evaluated on unseen models, yields rankings aligned with the target preferences, and it generalizes across model sizes from small to frontier scales using relatively limited training data. Through extensive experiments on the OpenLLMLeaderboard, BenchAlign outperforms existing refinement baselines (MetaBench, TinyBenchmarks, Random) in both pairwise accuracy and Spearman correlation across varying data availability, model sizes, and arbitrary model sets, with ablations confirming the value of learning-to-rank. The approach offers a practical, interpretable way to improve benchmark predictive utility and can accelerate model development toward real-world utility, while serving as a complementary tool to developing new benchmarks.

Abstract

Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance. However, many recent works find that benchmarks often fail to predict real utility. Towards bridging this gap, we introduce benchmark alignment, where we use limited amounts of information about model performance to automatically update offline benchmarks, aiming to produce new static benchmarks that predict model pairwise preferences in given test settings. We then propose BenchAlign, the first solution to this problem, which learns preference-aligned weight- ings for benchmark questions using the question-level performance of language models alongside ranked pairs of models that could be collected during deployment, producing new benchmarks that rank previously unseen models according to these preferences. Our experiments show that our aligned benchmarks can accurately rank unseen models according to models of human preferences, even across different sizes, while remaining interpretable. Overall, our work provides insights into the limits of aligning benchmarks with practical human preferences, which stands to accelerate model development towards real utility.

Aligning Language Model Benchmarks with Pairwise Preferences

TL;DR

The paper addresses the gap between benchmark performance and real-world utility by introducing benchmark alignment and BenchAlign, a method that learns preference-aligned weights for test items using a pairwise ranking loss to match a target human-preference ranking. BenchAlign builds a reweighted benchmark that, when evaluated on unseen models, yields rankings aligned with the target preferences, and it generalizes across model sizes from small to frontier scales using relatively limited training data. Through extensive experiments on the OpenLLMLeaderboard, BenchAlign outperforms existing refinement baselines (MetaBench, TinyBenchmarks, Random) in both pairwise accuracy and Spearman correlation across varying data availability, model sizes, and arbitrary model sets, with ablations confirming the value of learning-to-rank. The approach offers a practical, interpretable way to improve benchmark predictive utility and can accelerate model development toward real-world utility, while serving as a complementary tool to developing new benchmarks.

Abstract

Language model benchmarks are pervasive and computationally-efficient proxies for real-world performance. However, many recent works find that benchmarks often fail to predict real utility. Towards bridging this gap, we introduce benchmark alignment, where we use limited amounts of information about model performance to automatically update offline benchmarks, aiming to produce new static benchmarks that predict model pairwise preferences in given test settings. We then propose BenchAlign, the first solution to this problem, which learns preference-aligned weight- ings for benchmark questions using the question-level performance of language models alongside ranked pairs of models that could be collected during deployment, producing new benchmarks that rank previously unseen models according to these preferences. Our experiments show that our aligned benchmarks can accurately rank unseen models according to models of human preferences, even across different sizes, while remaining interpretable. Overall, our work provides insights into the limits of aligning benchmarks with practical human preferences, which stands to accelerate model development towards real utility.
Paper Structure (27 sections, 3 equations, 3 figures, 4 tables, 1 algorithm)

This paper contains 27 sections, 3 equations, 3 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Benchmark Alignment
  4. Problem Definition
  5. Proposed Method
  6. Experiments
  7. Experimental Setup
  8. Generalization Across Model Sizes
  9. Effects of Data Availability
  10. Generalization under Arbitrary Model Sets
  11. Ablation Studies
  12. Conclusions
  13. Reward Models as Proxies of Human Preferences
  14. Baselines
  15. MetaBench
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Overview of BenchAlign. Initially, we have a benchmark whose model ranking is not aligned with a target preference. BenchAlign takes benchmark performance and a target model ranking to learn a preference-aligned question reweighting. During inference, models evaluated on the reweighted benchmark are ranked consistently with the target preference.
  • Figure 2: Performance of BenchAlign under different number of models and learning-to-rank algorithms. Error bars obtained using the number of pairs and sample size from the holdout set for $Acc_{pair}$ and $\rho$, respectively.
  • Figure 3: Performance of BenchAlign under different numbers of questions and learning-to-rank algorithms. Error bars obtained using the number of pairs and sample size from the holdout set for $Acc_{pair}$ and $\rho$, respectively.