Best Arm Identification with LLM Judges and Limited Human

TL;DR

This work tackles fixed-confidence best-arm identification when each pull yields a cheap, biased proxy score $F(k,X)$ and an expensive ground-truth $Y(k,X)$ that is observed only through selective auditing. The authors introduce an IPW residual estimator that combines proxy means with inverse-propensity-weighted residuals to estimate arm means $\theta_k$, and they develop anytime-valid confidence sequences to enable valid sequential decisions under adaptive sampling and stopping. A two-loop algorithm (PP-LUCB) is proposed: an LUCB-style outer loop for arm selection and an inner loop that optimizes auditing probabilities, with a Neyman-style policy $\pi^*(x,f) \propto \sqrt{g(x,f)}$ guiding audit allocation to minimize variance under a budget. Theoretical guarantees include $\delta$-correctness and near-oracle audit efficiency, and empirical validation in synthetic settings shows high coverage (>$98\%$) and substantial cost reductions (up to $48\%$ over uniform auditing). The framework provides principled bias correction and efficient resource use for LLM-based judgments and other prediction-powered evaluation tasks in complex, bias-prone environments.

Abstract

We study fixed-confidence best-arm identification (BAI) where a cheap but potentially biased proxy (e.g., LLM judge) is available for every sample, while an expensive ground-truth label can only be acquired selectively when using a human for auditing. Unlike classical multi-fidelity BAI, the proxy is biased (arm- and context-dependent) and ground truth is selectively observed. Consequently, standard multi-fidelity methods can mis-select the best arm, and uniform auditing, though accurate, wastes scarce resources and is inefficient. We prove that without bias correction and propensity adjustment, mis-selection probability may not vanish (even with unlimited proxy data). We then develop an estimator for the mean of each arm that combines proxy scores with inverse-propensity-weighted residuals and form anytime-valid confidence sequences for that estimator. Based on the estimator and confidence sequence, we propose an algorithm that adaptively selects and audits arms. The algorithm concentrates audits on unreliable contexts and close arms and we prove that a plug-in Neyman rule achieves near-oracle audit efficiency. Numerical experiments confirm the theoretical guarantees and demonstrate the superior empirical performance of the proposed algorithm.

Figures (3)

• Figure 1: Anytime-valid coverage validation. Empirical coverage rates exceed theoretical $1-\delta$ targets across all confidence levels and sample sizes. Error bars show 95% bootstrap confidence intervals over 1000 trials. The polynomial stitched boundary confidence sequence maintains guaranteed coverage under adaptive sampling.
• Figure 2: Allocator performance across gap settings ($\Delta \in \{0.10, 0.15, 0.20\}$). Neyman allocation achieves 48--50% cost reduction over Uniform baseline while lying within 20% of Oracle upper bound. Error bars show standard deviations over 20 trials. The relative ordering of strategies is consistent across all gap values.
• Figure 3: Failure mode validation. Adaptive auditing (UncertaintyWeighted allocator) achieves 10% cost reduction over fixed auditing while maintaining 100% accuracy. No-Judge baseline requires full auditing (cost 10,500), while No-Audit fails completely (0% accuracy). The cost bars use log scale to accommodate the large range.

Theorems & Definitions (13)

• Definition 3.2: Judge-only algorithms
• Theorem 3.3: Judge-only impossibility
• Proposition 4.3: Anytime-valid CS for $\mu_{F,k}$
• Proposition 4.4: Anytime-valid CS for residual mean
• Theorem 4.5: Simultaneous anytime coverage for all arms
• Theorem 5.1: $\delta$-correct BAI
• Theorem 5.2: Oracle optimal auditing
• Theorem C.1: Sub-Gaussian CS, howard2021time Theorem 1
• Theorem C.2: Empirical Bernstein CS, howard2021time Theorem 4
• proof