Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Labels or Preferences? Budget-Constrained Learning with Human Judgments over AI-Generated Outputs

Zihan Dong, Ruijia Wu, Linjun Zhang

TL;DR

This work tackles budget-constrained learning where ground-truth labels, pairwise human preferences, and AI-generated pseudo-labels are combined to estimate a target parameter θ(P_{X,Y}). It casts budget allocation as a semiparametric, monotone missing-data problem under MAR and develops PCAL, an EIF-based method that jointly learns an optimal multi-type labeling policy α_j and constructs a statistically efficient estimator of θ. Theoretical results establish asymptotic normality and the semiparametric efficiency bound for PCAL, along with robustness guarantees to nuisance-model misspecification; a covariate-agnostic variant (MCAR) is also developed. Empirical evaluations on linear regression and a politeness analysis demonstrate substantial reductions in CI length while preserving coverage, validating practical efficiency gains under budget constraints. Overall, PCAL provides a principled, variance-minimizing framework for budget-aware data acquisition in modern AI systems.

Abstract

The increasing reliance on human preference feedback to judge AI-generated pseudo labels has created a pressing need for principled, budget-conscious data acquisition strategies. We address the crucial question of how to optimally allocate a fixed annotation budget between ground-truth labels and pairwise preferences in AI. Our solution, grounded in semi-parametric inference, casts the budget allocation problem as a monotone missing data framework. Building on this formulation, we introduce Preference-Calibrated Active Learning (PCAL), a novel method that learns the optimal data acquisition strategy and develops a statistically efficient estimator for functionals of the data distribution. Theoretically, we prove the asymptotic optimality of our PCAL estimator and establish a key robustness guarantee that ensures robust performance even with poorly estimated nuisance models. Our flexible framework applies to a general class of problems, by directly optimizing the estimator's variance instead of requiring a closed-form solution. This work provides a principled and statistically efficient approach for budget-constrained learning in modern AI. Simulations and real-data analysis demonstrate the practical benefits and superior performance of our proposed method.

Labels or Preferences? Budget-Constrained Learning with Human Judgments over AI-Generated Outputs

TL;DR

This work tackles budget-constrained learning where ground-truth labels, pairwise human preferences, and AI-generated pseudo-labels are combined to estimate a target parameter θ(P_{X,Y}). It casts budget allocation as a semiparametric, monotone missing-data problem under MAR and develops PCAL, an EIF-based method that jointly learns an optimal multi-type labeling policy α_j and constructs a statistically efficient estimator of θ. Theoretical results establish asymptotic normality and the semiparametric efficiency bound for PCAL, along with robustness guarantees to nuisance-model misspecification; a covariate-agnostic variant (MCAR) is also developed. Empirical evaluations on linear regression and a politeness analysis demonstrate substantial reductions in CI length while preserving coverage, validating practical efficiency gains under budget constraints. Overall, PCAL provides a principled, variance-minimizing framework for budget-aware data acquisition in modern AI systems.

Abstract

The increasing reliance on human preference feedback to judge AI-generated pseudo labels has created a pressing need for principled, budget-conscious data acquisition strategies. We address the crucial question of how to optimally allocate a fixed annotation budget between ground-truth labels and pairwise preferences in AI. Our solution, grounded in semi-parametric inference, casts the budget allocation problem as a monotone missing data framework. Building on this formulation, we introduce Preference-Calibrated Active Learning (PCAL), a novel method that learns the optimal data acquisition strategy and develops a statistically efficient estimator for functionals of the data distribution. Theoretically, we prove the asymptotic optimality of our PCAL estimator and establish a key robustness guarantee that ensures robust performance even with poorly estimated nuisance models. Our flexible framework applies to a general class of problems, by directly optimizing the estimator's variance instead of requiring a closed-form solution. This work provides a principled and statistically efficient approach for budget-constrained learning in modern AI. Simulations and real-data analysis demonstrate the practical benefits and superior performance of our proposed method.
Paper Structure (49 sections, 16 theorems, 213 equations, 7 figures, 5 algorithms)

This paper contains 49 sections, 16 theorems, 213 equations, 7 figures, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notation
  4. Organization
  5. Problem Formulation
  6. Data Setup
  7. Missing Data Framework
  8. Covariate-aware strategy
  9. Method
  10. Overview of semiparametric efficiency theory
  11. Efficient Influence Function
  12. Algorithms
  13. Covariate-agnostic strategy
  14. Theoretical Analysis
  15. Theoretical Guarantees of Propensity Score Estimator
  16. ...and 34 more sections

Key Result

Proposition 3.1

Consider the model space defined above. When the missingness mechanism satisfies MAR, the EIF of $\theta$ is given by: where and $\psi_j\qty((X, Y, W_1, W_2, V)^{r_j^*}) = \mathbb{E}\qty[\psi_{P_{X,Y}}(X, Y) \mid (X, Y, W_1, W_2, V)^{r_j^*}]$, $\gamma_j(X, W_1, W_2) = \sum_{l = 1}^j \alpha_l(X, W_1, W_2)$.

Figures (7)

  • Figure 1: Confidence interval results of $\theta_1^*$ with $c=10$. Results for $c \in \{5, 20\}$ are provided in Appendix \ref{['sec: app additional simulation figures']}.
  • Figure 2: Confidence interval results of feature 3 with $c=100$. Results for $c \in \{20, 50, 200\}$ are provided in Appendix \ref{['sec: app additional realdata figures']}.
  • Figure 3: Confidence interval results of $\theta_1^*$ with $c=5$.
  • Figure 4: Confidence interval results of $\theta_1^*$ with $c=20$.
  • Figure 5: Confidence interval results of feature 3 with $c=20$.
  • ...and 2 more figures

Theorems & Definitions (41)

  • Remark 2.1
  • Proposition 3.1
  • Remark 3.1
  • Remark 3.2
  • Remark 3.3
  • Theorem 3.1: Asymptotic Normality
  • Remark 3.4
  • Remark 3.5
  • Remark 3.6
  • Definition 4.1: Sub-Weibull Random Variable
  • ...and 31 more