Confident Rankings with Fewer Items: Adaptive LLM Evaluation with Continuous Scores
Esma Balkır, Alice Pernthaller, Marco Basaldella, José Hernández-Orallo, Nigel Collier
TL;DR
This work tackles the cost and statistical validity of evaluating generation-based LLM outputs by extending computerized adaptive testing to continuous bounded scores. It preserves the logistic mean structure while replacing the Bernoulli likelihood with a heteroskedastic normal: $\mu(\theta,b)=\frac{1}{1+e^{-(\theta-b)}}$ and $\sigma^2(\theta,b)=k\mu(1-\mu)$ with $k=1/a^2$, yielding Fisher Information $I(\theta|b,k)=\frac{\mu(1-\mu)}{k}$. Building on this, the authors introduce an adaptive multi-model ranking framework that stops testing when adjacent model pairs are statistically differentiated at level $\gamma$, and allocates tests to minimize cost via $\text{value}_m=\frac{\text{SE}_m^2}{(n_m+1)c_m}$. Empirically, across five generation benchmarks (e.g., GovReport, BioLaySumm2025, Nemotron PII, TruthfulQA, FLORES), the method achieves Kendall's $\tau=0.73$ relative to ground truth while using only $2\%$ of the full item budget, with substantial gains over random sampling and fixed-length CAT, and demonstrates robust transfer to unseen model families. The framework thus enables reliable, scalable comparisons of generation quality with explicit uncertainty quantification and cost-aware decision making, bearing practical impact for sustainable and fair model evaluation.
Abstract
Computerized Adaptive Testing (CAT) has proven effective for efficient LLM evaluation on multiple-choice benchmarks, but modern LLM evaluation increasingly relies on generation tasks where outputs are scored continuously rather than marked correct/incorrect. We present a principled extension of IRT-based adaptive testing to continuous bounded scores (ROUGE, BLEU, LLM-as-a-Judge) by replacing the Bernoulli response distribution with a heteroskedastic normal distribution. Building on this, we introduce an uncertainty aware ranker with adaptive stopping criteria that achieves reliable model ranking while testing as few items and as cheaply as possible. We validate our method on five benchmarks spanning n-gram-based, embedding-based, and LLM-as-judge metrics. Our method uses 2% of the items while improving ranking correlation by 0.12 τ over random sampling, with 95% accuracy on confident predictions.