Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment

Dario Loi, Elena Maria Muià, Federico Siciliano, Giovanni Trappolini, Vincenzo Crisà, Peter Kruger, Fabrizio Silvestri

TL;DR

AutoBench proposes a fully automated, self-sustaining framework for evaluating LLMs through reciprocal peer assessment, addressing the stagnation and contamination of static benchmarks. By allowing models to generate tasks, answer, and judge peers across diverse domains, and by aggregating judgments with an iterative weighting scheme, AutoBench yields dynamic consensus-based rankings. Empirical validation shows strong alignment with established benchmarks and a marked advantage for the multi-judge design over single-judge baselines, supporting robust, human-consistent evaluation without human supervision. This approach offers a scalable, contamination-resistant alternative for continuous evaluation as LLMs evolve, with potential implications for standardized leaderboards and ongoing benchmarking.

Abstract

We present AutoBench, a fully automated and self-sustaining framework for evaluating Large Language Models (LLMs) through reciprocal peer assessment. This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L.. Unlike static benchmarks that suffer from test-set contamination and limited adaptability, AutoBench dynamically generates novel evaluation tasks while models alternately serve as question generators, contestants, and judges across diverse domains. An iterative weighting mechanism amplifies the influence of consistently reliable evaluators, aggregating peer judgments into consensus-based rankings that reflect collective model agreement. Our experiments demonstrate strong correlations with established benchmarks including MMLU-Pro and GPQA (respectively 78\% and 63\%), validating this peer-driven evaluation paradigm. The multi-judge design significantly outperforms single-judge baselines, confirming that distributed evaluation produces more robust and human-consistent assessments. AutoBench offers a scalable, contamination-resistant alternative to static benchmarks for the continuous evaluation of evolving language models.

AutoBench: Automating LLM Evaluation through Reciprocal Peer Assessment

TL;DR

AutoBench proposes a fully automated, self-sustaining framework for evaluating LLMs through reciprocal peer assessment, addressing the stagnation and contamination of static benchmarks. By allowing models to generate tasks, answer, and judge peers across diverse domains, and by aggregating judgments with an iterative weighting scheme, AutoBench yields dynamic consensus-based rankings. Empirical validation shows strong alignment with established benchmarks and a marked advantage for the multi-judge design over single-judge baselines, supporting robust, human-consistent evaluation without human supervision. This approach offers a scalable, contamination-resistant alternative for continuous evaluation as LLMs evolve, with potential implications for standardized leaderboards and ongoing benchmarking.

Abstract

We present AutoBench, a fully automated and self-sustaining framework for evaluating Large Language Models (LLMs) through reciprocal peer assessment. This paper provides a rigorous scientific validation of the AutoBench methodology, originally developed as an open-source project by eZecute S.R.L.. Unlike static benchmarks that suffer from test-set contamination and limited adaptability, AutoBench dynamically generates novel evaluation tasks while models alternately serve as question generators, contestants, and judges across diverse domains. An iterative weighting mechanism amplifies the influence of consistently reliable evaluators, aggregating peer judgments into consensus-based rankings that reflect collective model agreement. Our experiments demonstrate strong correlations with established benchmarks including MMLU-Pro and GPQA (respectively 78\% and 63\%), validating this peer-driven evaluation paradigm. The multi-judge design significantly outperforms single-judge baselines, confirming that distributed evaluation produces more robust and human-consistent assessments. AutoBench offers a scalable, contamination-resistant alternative to static benchmarks for the continuous evaluation of evolving language models.
Paper Structure (15 sections, 3 equations, 3 figures, 8 tables)

This paper contains 15 sections, 3 equations, 3 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Experiments
  5. Experimental Setup
  6. Results and Discussion
  7. Conclusions and Future Work
  8. Limitations
  9. Appendix
  10. Prompts and Sampling Mechanisms
  11. Pseudocode
  12. Model Suite Details
  13. Supplementary Results
  14. LLM Usage Disclosure
  15. Computational Details

Figures (3)

  • Figure 1: Diagram of the AutoBench framework.
  • Figure 2: Convergence of weights over iterations, quantified by the L1 norm between successive evaluations.
  • Figure 3: Heatmap of the aggregate judgment matrix, averaged over all $T$ iterations, each cell $(r, c)$ shows the mean score assigned by judge $r$ to the contestant model $c$. The main diagonal $(r = c)$ reveals potential self-preference, as it shows the score each model gives to itself.