Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation

Qin Liu, Jacob Dineen, Yuxi Huang, Sheng Zhang, Hoifung Poon, Ben Zhou, Muhao Chen

TL;DR

The paper addresses the validity challenges of static, leakage-prone benchmarks for large language models by introducing ArenaBencher, a model-agnostic framework that evolves benchmarks through ability extraction, aligned variant generation, LLM-verified judgments, and multi-model feedback with iterative in-context refinement. It demonstrates that evolving benchmarks across math, safety, and commonsense domains yields harder, more discriminative, and fairer test items that retain alignment with original objectives. The approach mitigates overfitting to any single model and exposes shared failure modes, enabling more reliable cross-model comparisons in fast-moving AI landscapes. This scalable methodology offers contamination-resilient evaluation aligned with rapid foundation-model progress.

Abstract

Benchmarks are central to measuring the capabilities of large language models and guiding model development, yet widespread data leakage from pretraining corpora undermines their validity. Models can match memorized content rather than demonstrate true generalization, which inflates scores, distorts cross-model comparisons, and misrepresents progress. We introduce ArenaBencher, a model-agnostic framework for automatic benchmark evolution that updates test cases while preserving comparability. Given an existing benchmark and a diverse pool of models to be evaluated, ArenaBencher infers the core ability of each test case, generates candidate question-answer pairs that preserve the original objective, verifies correctness and intent with an LLM as a judge, and aggregates feedback from multiple models to select candidates that expose shared weaknesses. The process runs iteratively with in-context demonstrations that steer generation toward more challenging and diagnostic cases. We apply ArenaBencher to math problem solving, commonsense reasoning, and safety domains and show that it produces verified, diverse, and fair updates that uncover new failure modes, increase difficulty while preserving test objective alignment, and improve model separability. The framework provides a scalable path to continuously evolve benchmarks in step with the rapid progress of foundation models.

ArenaBencher: Automatic Benchmark Evolution via Multi-Model Competitive Evaluation

TL;DR

The paper addresses the validity challenges of static, leakage-prone benchmarks for large language models by introducing ArenaBencher, a model-agnostic framework that evolves benchmarks through ability extraction, aligned variant generation, LLM-verified judgments, and multi-model feedback with iterative in-context refinement. It demonstrates that evolving benchmarks across math, safety, and commonsense domains yields harder, more discriminative, and fairer test items that retain alignment with original objectives. The approach mitigates overfitting to any single model and exposes shared failure modes, enabling more reliable cross-model comparisons in fast-moving AI landscapes. This scalable methodology offers contamination-resilient evaluation aligned with rapid foundation-model progress.

Abstract

Benchmarks are central to measuring the capabilities of large language models and guiding model development, yet widespread data leakage from pretraining corpora undermines their validity. Models can match memorized content rather than demonstrate true generalization, which inflates scores, distorts cross-model comparisons, and misrepresents progress. We introduce ArenaBencher, a model-agnostic framework for automatic benchmark evolution that updates test cases while preserving comparability. Given an existing benchmark and a diverse pool of models to be evaluated, ArenaBencher infers the core ability of each test case, generates candidate question-answer pairs that preserve the original objective, verifies correctness and intent with an LLM as a judge, and aggregates feedback from multiple models to select candidates that expose shared weaknesses. The process runs iteratively with in-context demonstrations that steer generation toward more challenging and diagnostic cases. We apply ArenaBencher to math problem solving, commonsense reasoning, and safety domains and show that it produces verified, diverse, and fair updates that uncover new failure modes, increase difficulty while preserving test objective alignment, and improve model separability. The framework provides a scalable path to continuously evolve benchmarks in step with the rapid progress of foundation models.

Paper Structure

This paper contains 19 sections, 8 equations, 2 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. ArenaBencher
  4. Evaluation Target Extraction
  5. Candidate Generation and Verification
  6. Multi-Model Feedback Scoring
  7. Iterative Refinement with In-context Demonstration
  8. Final Selection and Benchmark Update
  9. Experiments and Evaluation
  10. Experimental Settings
  11. Main Results
  12. Conclusion and Future Work
  13. Conclusion and Future Work
  14. Appendix
  15. Prompts for ArenaBencher Components
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Overview of ArenaBencher on a math reasoning example. Starting from an original test case, the system extracts a structured target objective that specifies multiple rubrics. Conditioned on this objective and in-context demonstrations of strong candidates evaluated by multi-model feedback, the generator iteratively proposes multiple candidate queries and answers, and an independent LLM judge verifies correctness and test target alignment.
  • Figure 2: Case study of ArenaBencher-generated test case update. While the objective extraction succeeds, the updated test case generated by ArenaBencher fails for two key reasons. First, the updated question is not well-formed, as it omits necessary information, making it unsolvable. Second, although the updated query retains a similar surface-level objective, it introduces additional complexity by requiring a new mathematical operation (division), thus deviating from the original reasoning structure and increasing cognitive load.