Extending Prediction-Powered Inference through Conformal Prediction

TL;DR

The paper addresses the fragility of prediction-powered inference in the face of privacy, robustness, and distribution-shift guarantees by unifying it with conformal prediction. It introduces a general framework that imputes missing data via calibrated conformal set-predictors, thereby inheriting coverage-like guarantees and additional properties from conformal methods. The authors instantiate the framework for means, Z-/M-estimation, and e-value-based procedures, including offline e-values, with theoretical validity guarantees and practical power considerations. They demonstrate the approach on private healthcare data and online risk monitoring, showing improved or competitive performance relative to prior methods and establishing a new baseline for prediction-powered inference with guarantees.

Abstract

Prediction-powered inference is a recent methodology for the safe use of black-box ML models to impute missing data, strengthening inference of statistical parameters. However, many applications require strong properties besides valid inference, such as privacy, robustness or validity under continuous distribution shifts; deriving prediction-powered methods with such guarantees is generally an arduous process, and has to be done case by case. In this paper, we resolve this issue by connecting prediction-powered inference with conformal prediction: by performing imputation through a calibrated conformal set-predictor, we attain validity while achieving additional guarantees in a natural manner. We instantiate our procedure for the inference of means, Z- and M-estimation, as well as e-values and e-value-based procedures. Furthermore, in the case of e-values, ours is the first general prediction-powered procedure that operates off-line. We demonstrate these advantages by applying our method on private and time-series data. Both tasks are nontrivial within the standard prediction-powered framework but become natural under our method.

Figures (3)

• Figure 1: Our method is comparable to existing prediction-powered procedures. We conduct experiments on two datasets where previous prediction-powered methods are applicable: one on the prevalence of phishing attacks (a mean), and another on characterizing gene expression levels (a median). In (a) we see one realization of our CIs along with baselines, while in (b) we analyze the distribution of the interval sizes over varying data splits. In both cases our procedure outperforms only using labelled data, while edging over prior methods for the mean estimation task.
• Figure 2: Conformal prediction-powered inference with differential privacy. We apply our method to analyze the recurrence of thyroid cancer atop private patient data. With a single inference-agnostic and differentially-private calibration, we are able to do prediction-powered inference for the probabilities of recurrence for various strata, with minimal increase in interval size compared to a non-private calibration.
• Figure 3: Conformal prediction-powered continuous risk monitoring. Our method can also be naturally applied for continuous risk monitoring by simply using online conformal prediction methods for the calibration. It satisfies strong anytime-valid guarantees in a dynamic setting without requiring active data collection. The resulting procedure rejects nulls much more quickly than simply using labelled samples, attaining high statistical power.

Theorems & Definitions (33)

• Lemma 2.1
• Remark 2.2
• Proposition 2.3
• Proposition 2.4
• Proposition 2.5
• Proposition 2.6
• Proposition 2.7
• Proposition 2.8
• Proposition 2.9
• Lemma A.1: Lemma \ref{['thm:main-argument']} in the main text