Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

am-ELO: A Stable Framework for Arena-based LLM Evaluation

Zirui Liu, Jiatong Li, Yan Zhuang, Qi Liu, Shuanghong Shen, Jie Ouyang, Mingyue Cheng, Shijin Wang

TL;DR

This work tackles instability in arena-based LLM evaluation caused by data-order effects and annotator variability. It replaces the iterative ELO updates with a maximum-likelihood formulation (m-ELO) and extends the model to incorporate annotator abilities (am-ELO) using an IRT-inspired parameter $\theta_k$, with a normalization constraint to ensure identifiability. The authors prove concavity and uniqueness of the MLE solution and demonstrate, via real data and simulations, that am-ELO yields better fit, improved predictive performance, and enhanced detection of anomalous annotators, while substantially reducing score instability. The proposed Stable Arena Framework thus offers a robust, interpretable, and practical approach for ranking LLMs in arena settings, with clear guidance for annotator screening and reward mechanisms in crowdsourced evaluations.

Abstract

Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating's probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs.

am-ELO: A Stable Framework for Arena-based LLM Evaluation

TL;DR

This work tackles instability in arena-based LLM evaluation caused by data-order effects and annotator variability. It replaces the iterative ELO updates with a maximum-likelihood formulation (m-ELO) and extends the model to incorporate annotator abilities (am-ELO) using an IRT-inspired parameter , with a normalization constraint to ensure identifiability. The authors prove concavity and uniqueness of the MLE solution and demonstrate, via real data and simulations, that am-ELO yields better fit, improved predictive performance, and enhanced detection of anomalous annotators, while substantially reducing score instability. The proposed Stable Arena Framework thus offers a robust, interpretable, and practical approach for ranking LLMs in arena settings, with clear guidance for annotator screening and reward mechanisms in crowdsourced evaluations.

Abstract

Arena-based evaluation is a fundamental yet significant evaluation paradigm for modern AI models, especially large language models (LLMs). Existing framework based on ELO rating system suffers from the inevitable instability problem due to ranking inconsistency and the lack of attention to the varying abilities of annotators. In this paper, we introduce a novel stable arena framework to address these issues by enhancing the ELO Rating System. Specifically, we replace the iterative update method with a Maximum Likelihood Estimation (MLE) approach, m-ELO, and provide theoretical proof of the consistency and stability of the MLE approach for model ranking. Additionally, we proposed the am-ELO, which modify the Elo Rating's probability function to incorporate annotator abilities, enabling the simultaneous estimation of model scores and annotator reliability. Experiments demonstrate that this method ensures stability, proving that this framework offers a more robust, accurate, and stable evaluation method for LLMs.
Paper Structure (22 sections, 2 theorems, 17 equations, 7 figures, 2 tables, 3 algorithms)

This paper contains 22 sections, 2 theorems, 17 equations, 7 figures, 2 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Battle Scenario
  4. Annotator
  5. Ranking Systems for LLM Evaluation
  6. Preliminary
  7. ELO Rating System
  8. Better Performance estimation with ELO
  9. MLE for ELO (m-ELO) Estimation
  10. Convergence Analysis
  11. Annotator Modeling m-ELO (am-ELO) Estimation
  12. Normalization
  13. Stable Arena Framework
  14. Experiments
  15. Dataset
  16. ...and 7 more sections

Key Result

Theorem 4.1

Assume $R_0=0$ and $|S|$ is sufficiently large, then the log-likelihood function $\ln{L}$ with respect to $(R_2,\cdots,R_N)$ is a concave function and has at most one extreme point.

Figures (7)

  • Figure 1: An example of ELO score. The error bar represents the standard deviation and the error line represents the difference between the maximum or minimum value and the mean value. The line chart represents the ELO scores estimated from the records of the specific annotator.
  • Figure 2: The traditional iterative ELO method and our proposed am-ELO method based on MLE.
  • Figure 3: The result of each LLMs on different evaluation method. Specifically, the line chart represents the normalized ELO scores$\uparrow$ (ranging from 0 to 1) of each LLM under different evaluation methods. The bar chart represents the Loss$\downarrow$ (log-likelihood function) of each LLM's match records under different evaluation methods.
  • Figure 4: The heatmap shows the number of victories in battles between various models (Three models with similar abilities, koala-13b, vicuna-7b, gpt-13b, and the better or worse models than them). Each number in the figure represents the times the row model wins the column model in the battle.
  • Figure 5: The Loss and Consistency of the evaluation method at each epoch on the Chatbot dataset.
  • ...and 2 more figures

Theorems & Definitions (4)

  • Theorem 4.1
  • Theorem 4.2
  • proof
  • proof