Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

Mingyuan Xu, Xinzi Tan, Jiawei Wu, Doudou Zhou

TL;DR

The paper tackles evaluating LLMs on open-ended tasks without ground-truth labels by addressing judge reliability heterogeneity in LLM-as-a-judge rankings. It introduces a judge-aware Bradley–Terry–Luce model in which each judge $k$ has a discrimination parameter $\gamma_k$ and model scores are $(s_1,\dots,s_N)$, with $P(Y=1|i,j,k)=\sigma(\gamma_k(s_i-s_j))$, and identifiability achieved via normalization constraints. It establishes consistency and asymptotic normality: $\sqrt{T}(\hat{\boldsymbol{\theta}}-\boldsymbol{\theta}_0)\xrightarrow{d}\mathcal{N}(0,\boldsymbol{\Sigma}_{\boldsymbol{\theta}_0})$, enabling Wald intervals for score differences and ranks. Empirically, judge-aware aggregation improves agreement with human preferences and data efficiency on benchmarks and provides calibrated uncertainty, with a Chatbot Arena case recovering expected rankings and higher Spearman agreement with human judgments.

Abstract

Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates. More data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.

A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

TL;DR

The paper tackles evaluating LLMs on open-ended tasks without ground-truth labels by addressing judge reliability heterogeneity in LLM-as-a-judge rankings. It introduces a judge-aware Bradley–Terry–Luce model in which each judge has a discrimination parameter and model scores are , with , and identifiability achieved via normalization constraints. It establishes consistency and asymptotic normality: , enabling Wald intervals for score differences and ranks. Empirically, judge-aware aggregation improves agreement with human preferences and data efficiency on benchmarks and provides calibrated uncertainty, with a Chatbot Arena case recovering expected rankings and higher Spearman agreement with human judgments.

Abstract

Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates. More data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.
Paper Structure (18 sections, 3 theorems, 10 equations, 4 figures, 2 tables)

This paper contains 18 sections, 3 theorems, 10 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Statistical Models for Pairwise Ranking
  4. Judge Reliability in Pairwise Ranking
  5. Ranking Large Language Models
  6. Model and Method
  7. Problem Setup and Model
  8. Maximum Likelihood Estimation
  9. Asymptotic Properties
  10. Simulation Study
  11. Convergence Rate of the MLE
  12. Coverage and Width of Confidence Intervals
  13. Experiments on LLM Ranking Benchmarks
  14. Experimental Setup
  15. Alignment with Established Benchmarks
  16. ...and 3 more sections

Key Result

Theorem 3.2

Suppose Assumption assmp:conditions holds. If two parameter pairs $(\mathbf{s},\boldsymbol{\gamma})$ and $(\tilde{\mathbf{s}},\tilde{\boldsymbol{\gamma}})$ satisfy then there exist constants $a \in \mathbb{R}\setminus\{0\}$ and $b \in \mathbb{R}$ such that $\tilde{\mathbf{s}} = a \mathbf{s} + b \mathbf{1}$ and $\tilde{\boldsymbol{\gamma}} = \boldsymbol{\gamma} / a$. Conversely, any such transform

Figures (4)

  • Figure 1: MSE versus the number of comparisons $T$ for four configurations $(N,K)\in\{(10,5),(20,10),(50,20),(100,20)\}$. Both axes are logarithmic. Solid lines show means over $R=100$ runs, and shaded bands indicate interquartile ranges.
  • Figure 2: Empirical coverage probability (top) and average confidence-interval width (bottom) versus $T$ for four configurations $(N,K)\in\{(10,5),(20,10),(50,20),(100,20)\}$. In the top panel, the $x$-axis is logarithmic; in the bottom panel, both axes are logarithmic. Each point is based on $B=500$ replications.
  • Figure 3: UltraFeedback preference scores versus $\widehat{\mathbf{s}}$ across four evaluation dimensions (helpfulness, truthfulness, honesty, and instruction-following). Each point corresponds to a model; higher fitted scores are associated with higher preference ratings across all dimensions.
  • Figure 4: Average correlation of estimated s to the ground truth across different numbers of judges and sample sizes.

Theorems & Definitions (3)

  • Theorem 3.2: Identifiability
  • Corollary 3.3: Uniqueness under normalization
  • Theorem 3.4: Consistency and asymptotic normality