Measuring all the noises of LLM Evals

TL;DR

This work addresses how to robustly evaluate LLMs by decomposing evaluation noise into prediction and data components using the law of total variance. It introduces the all-pairs paired method to measure these noise components across all model pairs and evals, enabling accurate noise estimation and power analyses with formulas such as $\mathrm{Var}_{x,\epsilon}[A] = \mathrm{Var}_x[\mathrm{E}_{\epsilon}[A]] + \mathrm{E}_x[\mathrm{Var}_{\epsilon}[A]]$. The authors report two main findings: total noise is predictable as a function of model accuracy and prediction noise typically dominates data noise, implying that averaging predictions can dramatically improve statistical power. They provide estimators, small-$K$ corrections, and practical guidance for noise-aware meta-analysis of multiple evals, offering a path to more robust and scalable evaluation of LLMs.

Abstract

Separating signal from noise is central to experimental science. Applying well-established statistical method effectively to LLM evals requires consideration of their unique noise characteristics. We clearly define and measure three types of noise: prediction noise from generating different answers on a given question, data noise from sampling questions, and their combined total noise following the law of total variance. To emphasize relative comparisons and gain statistical power, we propose the all-pairs paired method, which applies the paired analysis to all pairs of LLMs and measures all the noise components based on millions of question-level predictions across many evals and settings. These measurements revealed clear patterns. First, each eval exhibits a characteristic and highly predictable total noise level across all model pairs. Second, paired prediction noise typically exceeds paired data noise, which means reducing prediction noise by averaging can significantly increase statistical power. These findings enable practitioners to assess significance without custom testing and to detect much smaller effects in controlled experiments.

Figures (6)

• Figure 1: paired total standard errors vs. the accuracy, showing a clear trend in the empirical results agreeing with the beta theory prediction. Left: on SWEbench-verified 1 prediction per question so only the total noise is estimated, Right: MATH500 with 1000 predictions per question and estimated data noise SE_x, the prediction noise SE_pred, and total noise SE. More details in §\ref{['sec:experiment']}.
• Figure 2: Increasingly sensitive training curves on SWEBench-Verified with $N=500$, A and B both start from the same checkpoint at step 0. Column 1) Original training curve. One prediction accuracy, showing B is better than A over all steps. 2) Unpaired bootstrap. A is better than B on many resampled set of 500 questions, showing a meaningless difference using this analysis method. 3) Paired bootstrap. Slightly significant with a $z$-score of 1.7 at 1 prediction per question. 4) Paired, averaged bootstrap. The result is very significant with a $z$-score of 3.5 when the question level predictions are averaged over 5 consecutive training checkpoints to reduce the prediction noise.
• Figure 3: Relative errors of the paired variances. $K=5$ samples are drawn from the 500 questions from MATH500, where 1000 real predictions are drawn from models $A$ and $B$ on each question, which is treated as the ground truth. The root mean squared relative errors (rms) of variance components are plotted vs. bootstrap sample size $N$, extending beyond the population size 500. The left figure without the correction on $K$ has an unacceptably high 70% relative error even with 2000 data points. The middle figure is with the correction of §\ref{['sec:sample_estimator']} and the right figure uses $1/K$ instead of $1/(K-1)$.
• Figure 4: Results heatmap. Each row is a different question, sorted from the easiest to the hardest. Each column is a model whose $x$-coordinate is the overall accuracy. Left: SWEbench-verified, 1 prediction per question; Right: MATH500, 1000 predictions per question. See §\ref{['sec:exploratorytools']} for more details.
• Figure 5: All noise components on CRUXEval at temperature 0.8 (left) and 0.2 (right). The prediction noise dominates at 0.8 temperature, whereas the data noise dominates at 0.2 temperature, while both still yielding about the same total noise.

Theorems & Definitions (7)

• Example 1: HumanEval
• Example 2
• Example 3: Paired vs unpaired
• Example 4: Error bars for leaderboards
• Example 5: majority voting
• Example 6: reliable verification
• Example 7: Expected metric and alternatives