Efficient Inference for Noisy LLM-as-a-Judge Evaluation

TL;DR

This work addresses the challenge of evaluating generative AI outputs when LLM judges imperfectly reflect human truth. It casts LLM-as-a-judge evaluation as a measurement-error problem and unifies two main approaches—Rogan–Gladen misclassification correction and surrogate-outcome inference (PPI/PPI++)—through semiparametric efficiency theory. The authors derive an efficient influence function (EIF)–based estimator, prove its efficiency, and show that in the binary outcome setting optimally tuned PPI++ matches EIF (and MLE), outperforming RG in finite samples. Through extensive simulations and a real-data application, they demonstrate that EIF/PPI++ provide valid uncertainty quantification with substantially narrower confidence intervals than RG, while remaining robust to judge quality and calibration budget. The results offer a principled, practical framework for reliable calibration and uncertainty quantification in LLM-as-a-judge evaluations, with code and benchmarks openly available for practitioners.

Abstract

Large language models (LLMs) are increasingly used as automatic evaluators of generative AI outputs, a paradigm often referred to as "LLM-as-a-judge." In practice, LLM judges are imperfect predictions for the underlying truth and can exhibit systematic, non-random errors. Two main approaches have recently been proposed to address this issue: (i) direct measurementerror correction based on misclassification models such as Rogan-Gladen-style estimators, and (ii) surrogate-outcome approaches such as prediction-powered inference (PPI), which correct bias by calibrating prediction residuals on a small set of gold-standard human labels. In this paper, we systematically study the performance of these two approaches for estimating mean parameters (e.g., average benchmark scores or pairwise win rates). Leveraging tools from semiparametric efficiency theory, we unify the two classes of estimators by deriving explicit forms of efficient influence function (EIF)-based efficient estimators and characterize conditions under which PPI-style estimators attain strictly smaller asymptotic variance than measurement-error corrections. We verify our theoretical results in simulations and demonstrate the methods on real-data examples. We provide an implementation of the benchmarked methods and comparison utilities at https://github.com/yiqunchen/debias-llm-as-a-judge.

Figures (9)

• Figure 1: Calibrating LLM-as-a-judge evaluations.(A) LLM-generated labels $\hat{Y}$ are noisy, with sensitivity $q_1<1$ and specificity $q_0<1$. (B) A human-labeled calibration set enables bias correction using PPI, Rogan--Gladen (RG), or EIF-based estimators. (C) Comparison of naive, RG, PPI, and EIF-based estimators for evaluating LLM-as-a-judge performance.
• Figure 2: Estimator bias of $\hat{\theta}$. All debiased estimators achieve near-zero bias; the naive estimator (red) exhibits large bias in many settings. When only 1% of the data is labeled, $\hat{\theta}_{\mathrm{RG}}$ also exhibits considerable bias.
• Figure 3: Empirical coverage rates under the symmetric case $q_0 = q_1$. The naive estimator exhibits severe undercoverage. RG, PPI, and PPI++ generally achieve nominal coverage; however, RG tends to overcover when the labeled fraction $m/n$ is small and judge quality is low (i.e., smaller $q_0 + q_1$).
• Figure 4: Mean confidence interval width for the symmetric case $q_0 = q_1$. EIF and PPI++ produce nearly identical and shortest intervals, outperforming Rogan--Gladen by a factor of 3--15$\times$ depending on the labeling ratio. The advantage is most pronounced when $q_0 + q_1 - 1$ is small (i.e., when the LLM-judge is closer to random guessing). MLE produces slightly wider intervals but achieves coverage closer to nominal. Naive confidence intervals are excluded since they are too narrow and do not achieve the desired coverage.
• Figure 5: Coverage (left) and CI width (right) for discrete prediction simulations. The dashed line indicates nominal 90% coverage. Most methods achieve approximately valid coverage across varying bias magnitudes ($\mu_3$), with naive intervals severely undercovering as bias increases. EIF with spline and GAM calibration achieves the narrowest intervals, reflecting its ability to correct discrete predictions. As predicted, linear correction and PPI++ tread closely.

Theorems & Definitions (18)

• Proposition 1: Asymptotic distribution of $\hat{\theta}_{\mathrm{RG}}$ in \ref{['eq:theta-RG']}
• Proposition 2
• Proposition 3
• Proposition 4: Asymptotic normality and efficiency of the MLE
• Proposition 5: Efficient influence function
• Proposition 6
• Proposition 7
• proof
• proof
• proof