Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions

Ruochen Zhao, Wenxuan Zhang, Yew Ken Chia, Weiwen Xu, Deli Zhao, Lidong Bing

TL;DR

Auto-Arena introduces a fully automatic LLM evaluation framework that uses an examiner to generate questions, two LLMs to engage in multi-round peer battles, and a committee of LLM judges to decide winners. The approach emphasizes dynamic questioning, adversarial debate, and collaborative judging to surface deeper capabilities and reduce bias. In experiments with 15 LLMs, Auto-Arena achieves state-of-the-art alignment with human preferences (96+% of human-consensus level) while eliminating manual data collection and judgments. The framework is validated across languages and domains, and it reveals interesting LLM behaviors such as competitive strategies and learning from opponents, suggesting promising directions for automated, scalable, and fair LLM evaluation.

Abstract

As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly. Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena. However, human evaluations require significant manual effort. To address this, we propose the Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents. Firstly, an LLM examiner generates questions. Then, two LLM candidates engage in a multi-round peer battle based on individual questions, aiming at revealing their true performance differences. Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness. During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive behaviors and even learn from the opponents. In our extensive experiments involving 15 recent LLMs, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks without any manual efforts. As a result, Auto-Arena offers a promising alternative to current human evaluation platforms for evaluating LLMs automatically.

Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions

TL;DR

Auto-Arena introduces a fully automatic LLM evaluation framework that uses an examiner to generate questions, two LLMs to engage in multi-round peer battles, and a committee of LLM judges to decide winners. The approach emphasizes dynamic questioning, adversarial debate, and collaborative judging to surface deeper capabilities and reduce bias. In experiments with 15 LLMs, Auto-Arena achieves state-of-the-art alignment with human preferences (96+% of human-consensus level) while eliminating manual data collection and judgments. The framework is validated across languages and domains, and it reveals interesting LLM behaviors such as competitive strategies and learning from opponents, suggesting promising directions for automated, scalable, and fair LLM evaluation.

Abstract

As LLMs continuously evolve, there is an urgent need for a reliable evaluation method that delivers trustworthy results promptly. Currently, static benchmarks suffer from inflexibility and unreliability, leading users to prefer human voting platforms like Chatbot Arena. However, human evaluations require significant manual effort. To address this, we propose the Auto-Arena, an innovative framework that automates the entire evaluation process using LLM-powered agents. Firstly, an LLM examiner generates questions. Then, two LLM candidates engage in a multi-round peer battle based on individual questions, aiming at revealing their true performance differences. Finally, a committee of LLM judges collaboratively discusses and decides the winner, reducing bias and enhancing fairness. During the peer battles, we observe intriguing scenarios where the LLM candidates display competitive behaviors and even learn from the opponents. In our extensive experiments involving 15 recent LLMs, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks without any manual efforts. As a result, Auto-Arena offers a promising alternative to current human evaluation platforms for evaluating LLMs automatically.
Paper Structure (34 sections, 11 figures, 12 tables)

This paper contains 34 sections, 11 figures, 12 tables.

Table of Contents

  1. Introduction
  2. The Auto-Arena Framework
  3. Question Generation
  4. Peer Debate
  5. Committee Discussions
  6. Using Auto-Arena to Derive Trustworthy Rankings
  7. Experimental Setup
  8. Results: Alignment with Human Preferences
  9. Ablation Studies on Peer Battles and Committee Discussions
  10. Constructing and Maintaining a Leaderboard with Auto-Arena
  11. Update New Models to Leaderboard
  12. Easy Extension to Other Domains and Languages
  13. Investigation of LLM's Behaviors in Competitive Peer Battles
  14. Peer Battles Make the Performance Gaps Become Visible
  15. LLMs Can Skillfully Attack the Opponents
  16. ...and 19 more sections

Figures (11)

  • Figure 1: An illustration of Auto-Arena.
  • Figure 2: The process of a Lincoln-Douglas-style peer battle with the actions used. The <Think> action can be used by the candidates freely and is only visible to the candidate itself.
  • Figure 3: Cohen’s Kappa agreement with majority vote results before (upper) and after (lower) committee discussions.
  • Figure 4: Changes in Elo scores of adding Llama-3 to the ranking of 9 models.
  • Figure 5: Elo scores of 15 models by Auto-Arena on English.
  • ...and 6 more figures