Generalized Prediction-Powered Inference, with Application to Binary Classifier Evaluation

TL;DR

This paper generalizes Prediction-Powered Inference (PPI) from M-estimators to any regular ALE, enabling rectified inference using unlabeled data to reduce variance while remaining computationally simple. It integrates missing-data and semiparametric efficiency insights, showing PPI does not generally attain the semiparametric efficiency bound but can serve as a practical alternative, and extends the method to handle three covariate-shift targets via IPW, IOW, and their combinations. The framework is specialized to binary classifier metrics (TPR, FPR, AUC) and validated through extensive simulations and a wine-quality case study, demonstrating variance reductions and reliable coverage when a useful predictions model is available. Collectively, the approach offers a scalable, flexible toolkit for improving inference under partially labeled data and distribution shift without requiring full derivations of efficient influence functions.

Abstract

In the partially-observed outcome setting, a recent set of proposals known as "prediction-powered inference" (PPI) involve (i) applying a pre-trained machine learning model to predict the response, and then (ii) using these predictions to obtain an estimator of the parameter of interest with asymptotic variance no greater than that which would be obtained using only the labeled observations. While existing PPI proposals consider estimators arising from M-estimation, in this paper we generalize PPI to any regular asymptotically linear estimator. Furthermore, by situating PPI within the context of an existing rich literature on missing data and semi-parametric efficiency theory, we show that while PPI does not achieve the semi-parametric efficiency lower bound outside of very restrictive and unrealistic scenarios, it can be viewed as a computationally-simple alternative to proposals in that literature. We exploit connections to that literature to propose modified PPI estimators that can handle three distinct forms of covariate distribution shift. Finally, we illustrate these developments by constructing PPI estimators of true positive rate, false positive rate, and area under the curve via numerical studies.

Figures (11)

• Figure 1: Ratio of estimated standard error for the PPI estimator to that of the estimator using only labeled data, averaged over 2500 replications, with $n=1000,$$\lambda\in\left\{0.01,0.1,0.25,0.5,0.8\right\}$, for mean, TPR, FPR, and AUC estimation. Shapes and colors denote the different prediction models $f$ used in PPI. The standard errors of PPI estimators are no greater than those of the estimators using only the labeled data. The ideal model, $\Pr(Y=1\mid \textbf{X})$ (squares), provides the most improvement for every estimand. The noisy prediction models, $\text{RF}(Y\sim X_4+X_5)$ (diamonds) and $\text{Unif}[0.01,0.99]$ (crossed lines), have no improvement for mean or AUC estimation, but do result in a variance decrease in TPR and FPR estimation.
• Figure 2: Empirical coverage versus nominal coverage, averaged over 2500 replications, with $n=1000$, $\lambda=0.1$, in the context of mean, TPR, FPR, and AUC estimation. Shapes denote different prediction models $f$ used in PPI. The red dashed line is the diagonal line. The empirical coverages are close to the nominal coverages.
• Figure 3: Ratio of estimated standard error for the PPI estimator to that of the estimator $\hat{\theta}_{\text{lab}}$ defined in Section \ref{['sec:covdistall']}, averaged over 2500 replications, with $n+N=50,000$, for mean, TPR, FPR, and AUC estimation, in the settings of Section \ref{['sec:covdistall']}. Shapes denote different prediction models $f$ used in PPI. The standard errors of the PPI estimators are no greater than those of $\hat{\theta}_{\text{lab}}$. The ideal model, $\Pr(Y=1\mid \textbf{X})$ (squares), provides the most improvement for every estimand. The noisy prediction models, $\text{RF}(Y\sim X_4+X_5)$ (diamonds) and $\text{Unif}[0.01,0.99]$ (crossed lines), have no improvement for mean or AUC estimation, but do result in a variance decrease in TPR and FPR estimation.
• Figure 4: Empirical coverage versus nominal coverage, averaged over 2500 replications, with $n+N=50,000$, for mean, TPR, FPR, and AUC estimation, in the setting of Section \ref{['sec:covdistall']}. Shapes denote different prediction models $f$ used in PPI. The red dashed line is the diagonal line. The empirical coverages are close to the nominal coverage.
• Figure S.1: Ratio of estimated standard error for the PPI estimator to that of the estimator using only labeled data, averaged over 2500 replications, with $n=1000$, $\lambda=0.1$, for TPR, FPR, and AUC estimation (shapes), in a setting where $R=f(X)$.

Theorems & Definitions (23)

• Theorem 2.1: Improving $\hat{\theta}_n$ with a rectifier
• Proposition 2.2: Asymptotically linear PPI
• Proposition 2.3
• Remark 3.1.1
• Theorem 3.1: PPI for targets of the full data distribution
• Remark 3.1.2
• Remark 3.2.1
• Theorem 3.2: PPI for targets of the unlabeled data distribution
• Remark 3.3.1
• Theorem 3.3: PPI for targets of the labeled data distribution