Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Drawing Conclusions from Draws: Rethinking Preference Semantics in Arena-Style LLM Evaluation

Raphael Tang, Crystina Zhang, Wenyan Li, Carmen Lai, Pontus Stenetorp, Yao Lu

TL;DR

This paper questions the standard interpretation that draws in arena-style LLM evaluations denote equal model ability and rating parity. It hypothesizes that draws largely reflect query properties, particularly difficulty and objectivity, and tests this across three real-world datasets with four rating systems. Key findings show that skipping draw updates improves prequential battle prediction accuracy by about 0.5–3.0%, and that draws are more common on easy or highly objective queries with risk ratios around 1.35–1.37. The work advocates rethinking draw semantics to incorporate query properties in rating updates and provides open-source code for replication.

Abstract

In arena-style evaluation of large language models (LLMs), two LLMs respond to a user query, and the user chooses the winning response or deems the "battle" a draw, resulting in an adjustment to the ratings of both models. The prevailing approach for modeling these rating dynamics is to view battles as two-player game matches, as in chess, and apply the Elo rating system and its derivatives. In this paper, we critically examine this paradigm. Specifically, we question whether a draw genuinely means that the two models are equal and hence whether their ratings should be equalized. Instead, we conjecture that draws are more indicative of query difficulty: if the query is too easy, then both models are more likely to succeed equally. On three real-world arena datasets, we show that ignoring rating updates for draws yields a 1-3% relative increase in battle outcome prediction accuracy (which includes draws) for all four rating systems studied. Further analyses suggest that draws occur more for queries rated as very easy and those as highly objective, with risk ratios of 1.37 and 1.35, respectively. We recommend future rating systems to reconsider existing draw semantics and to account for query properties in rating updates.

Drawing Conclusions from Draws: Rethinking Preference Semantics in Arena-Style LLM Evaluation

TL;DR

This paper questions the standard interpretation that draws in arena-style LLM evaluations denote equal model ability and rating parity. It hypothesizes that draws largely reflect query properties, particularly difficulty and objectivity, and tests this across three real-world datasets with four rating systems. Key findings show that skipping draw updates improves prequential battle prediction accuracy by about 0.5–3.0%, and that draws are more common on easy or highly objective queries with risk ratios around 1.35–1.37. The work advocates rethinking draw semantics to incorporate query properties in rating updates and provides open-source code for replication.

Abstract

In arena-style evaluation of large language models (LLMs), two LLMs respond to a user query, and the user chooses the winning response or deems the "battle" a draw, resulting in an adjustment to the ratings of both models. The prevailing approach for modeling these rating dynamics is to view battles as two-player game matches, as in chess, and apply the Elo rating system and its derivatives. In this paper, we critically examine this paradigm. Specifically, we question whether a draw genuinely means that the two models are equal and hence whether their ratings should be equalized. Instead, we conjecture that draws are more indicative of query difficulty: if the query is too easy, then both models are more likely to succeed equally. On three real-world arena datasets, we show that ignoring rating updates for draws yields a 1-3% relative increase in battle outcome prediction accuracy (which includes draws) for all four rating systems studied. Further analyses suggest that draws occur more for queries rated as very easy and those as highly objective, with risk ratios of 1.37 and 1.35, respectively. We recommend future rating systems to reconsider existing draw semantics and to account for query properties in rating updates.

Paper Structure

This paper contains 11 sections, 9 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Arena-Style Evaluation
  3. Preliminaries
  4. Rating Systems
  5. Experiments
  6. Draw Update Ablation Study
  7. Draw Semantics Study
  8. Related Work
  9. Conclusions
  10. Further Ablation
  11. Example User Interface

Figures (4)

  • Figure 1: The risk ratio of observing a draw compared to a win or a loss, binned by difficulty and subjectivity. Error bars denote 95% confidence intervals.
  • Figure 2: The risk ratio of observing a draw as a function of the absolute difference in model ratings, binned by percentile range.
  • Figure 3: Trade-off curves between draw and win/loss prediction accuracy as we vary the draw margin. Larger draw margins result in better draw prediction accuracy at the expense of win/loss accuracy, and vice versa. Curves with higher maxima and AUC are better.
  • Figure 4: An example user interface from https://lmarena.ai.