How Many Human Judgments Are Enough? Feasibility Limits of Human Preference Evaluation

TL;DR

The paper addresses how many human judgments are needed to reliably detect small improvements in open-ended generative tasks, and introduces a KL-divergence budget framework to quantify feasibility across prompt heterogeneity. It develops a minimax theory showing that when signal is diffuse across prompts, proportional allocation is minimax-optimal and adaptive allocation cannot substantially beat the budget limit; adaptive gains are possible only when signal concentrates in a small subset of prompts. The authors validate the theory empirically across Chatbot Arena, MT-Bench, image and code-generation datasets, finding many comparisons lie in near-tie regimes that require hundreds to thousands of judgments for reliable detection, and showing protocol design (curated vs open-ended) substantially shapes detectability through prompt-level variance. The work provides practical guidance for evaluation design, including pilot studies to estimate effect size, closed-form budgeting (n ≈ 2.63/δ²), and strategies like two-stage allocation when signal concentrates, to avoid underpowered conclusions and to calibrate budgets to expected perceptual gains.

Abstract

Human preference evaluations are widely used to compare generative models, yet it remains unclear how many judgments are required to reliably detect small improvements. We show that when preference signal is diffuse across prompts (i.e., all prompt types are similarly informative), proportional allocation is minimax-optimal: no allocation strategy substantially improves detectability. Empirical analysis of large-scale human preference datasets shows that most comparisons fall into this diffuse regime, exhibiting small preference margins that require far more judgments than typically collected, even in well-sampled comparisons. These limits persist across evaluation protocols and modalities, including chat, image generation, and code generation with execution feedback. In contrast, curated benchmarks that reduce prompt induced variability systematically induce larger margins and improve detectability through a $1.5\times$ reduction in prompt-level variance. Our results show that inconclusive or negative human evaluation outcomes frequently reflect underpowered evaluation rather than model equivalence, underscoring the need to account explicitly for effect size, budget, and protocol design.

Figures (9)

• Figure 1: Distribution of empirical absolute preference margins $|\delta|$ for well-sampled Chatbot Arena model pairs ($\geq$200 decisive judgments). A substantial fraction lie in a low-signal regime ($|\delta| \lesssim 0.10$), where reliable detection requires hundreds of judgments.
• Figure 2: Empirical validation of theoretical scaling. Each point corresponds to a model comparison, plotted by its preference margin and the implied evaluation budget required for detection power. The solid curve shows the predicted asymptotic $\delta^{-2}$ scaling from Eq. \ref{['eq:closed_form_budget']}.
• Figure 3: Distribution of absolute preference margins $|\delta|$ under Chatbot Arena and MT-Bench. Violin plots show margins for all model pairs meeting the sampling criteria of each protocol (Arena: $n=34$ pairs with $\geq$200 judgments; MT-Bench: $n=15$ pairs). The dashed line marks the near-tie threshold ($|\delta| = 0.10$), below which reliable detection typically requires hundreds of human judgments ($\approx$300 for 90% power). MT-Bench exhibits a shift toward larger preference margins, concentrating more mass in higher-signal regimes associated with lower evaluation budgets.
• Figure 4: Empirical power as a function of total evaluation budget $B$. Two-stage allocation improves detectability when signal is concentrated across prompt types (left), but underperforms proportional allocation in the diffuse regime (right).
• Figure 5: Screening accuracy measured by Jaccard overlap with the oracle top-$q$ set. Exact recovery of high-signal prompt types is limited even in the concentrated regime, indicating that screening accuracy alone does not explain the gains of Algorithm \ref{['alg:two-stage']}.

Theorems & Definitions (12)

• Lemma 3.1: Quadratic KL scaling
• Theorem 3.2: Minimax lower bound
• Theorem 3.3: Matching upper bound
• Proposition 3.4
• Theorem 3.5: Minimax optimality of proportional allocation
• Theorem 3.6: Adaptive gains under stochastic heterogeneity
• Corollary 3.7: Practical parameter choices for two-stage allocation
• proof
• Lemma 2.1: LLR concentration under $P_{\delta}$
• proof