CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control

TL;DR

This work tackles online selective predictive inference by introducing CAP, Calibration after Adaptive Pick, which adaptively selects calibration data from history to build exchangeable calibration sets for conformal prediction. CAP achieves finite-sample, distribution-free selection-conditional coverage and real-time FCR control, and it can be embedded in dynamic conformal prediction to handle distribution shifts. The authors develop adaptive and nonadaptive calibration strategies across decision-driven and symmetric-threshold online selections, prove SCC and FCR guarantees, and extend the framework to online multiple testing. Extensive synthetic and real-data experiments (including drug discovery and stock volatility) show CAP yields narrower prediction intervals while maintaining target FCR, outperforming baselines like LORD-CI and standard online CP under various settings.

Abstract

We study the problem of post-selection predictive inference in an online fashion. To avoid devoting resources to unimportant units, a preliminary selection of the current individual before reporting its prediction interval is common and meaningful in online predictive tasks. Since the online selection causes a temporal multiplicity in the selected prediction intervals, it is important to control the real-time false coverage-statement rate (FCR) which measures the overall miscoverage level. We develop a general framework named CAP (Calibration after Adaptive Pick) that performs an adaptive pick rule on historical data to construct a calibration set if the current individual is selected and then outputs a conformal prediction interval for the unobserved label. We provide tractable procedures for constructing the calibration set for popular online selection rules. We proved that CAP can achieve an exact selection-conditional coverage guarantee in the finite-sample and distribution-free regimes. To account for the distribution shift in online data, we also embed CAP into some recent dynamic conformal prediction algorithms and show that the proposed method can deliver long-run FCR control. Numerical results on both synthetic and real data corroborate that CAP can effectively control FCR around the target level and yield more narrowed prediction intervals over existing baselines across various settings.

Figures (18)

• Figure 1: The workflow of CAP at time $t$. The picked calibration set is $\{(X_s,Y_s)\}_{s\in \widehat{{\mathcal{C}}}_t}$, where $\widehat{{\mathcal{C}}}_t = \left\{s\in {\mathcal{H}}_t: \Pi_{t,s}^{\rm{Ada}}(X_s) = 1\right\}$. The residuals are computed by $R_s = |\widehat{\mu}(X_s) - Y_s|$.
• Figure 2: Plot for the real-time PIs for selected points from time 800 to 900. The selected points are marked by the cross. The experimental setup is the same as Scenario B with a decision-driven selection rule in Section \ref{['sec:experiments']}. The PIs are constructed by three methods with a target FCR level $10\%$. Red interval: CAP (FCP at index 900 is $7.62\%$); Blue interval: ordinary online conformal prediction which provides marginal interval (FCP is $20.95\%$); Orange interval: LORD-CI with defaulted parameters (FCP is $1.59\%$). Points circled by hollow diamond symbols indicate cases where CAP successfully covers the true response, while OCP fails.
• Figure 3: Real-time FCR and average length from time $20$ to $1,000$ for different scenarios and selection rules. The black dashed line denotes the target FCR level $10\%$.
• Figure 4: Comparison for CAP-DtACI, CAP and DtACI by real-time FCR and average length from time $100$ to $2,000$ for quantile selection rule under different data-generating settings. The black dashed line represents the target FCR level $10\%$.
• Figure 5: Real-time FCR and average length from time $20$ to $2,000$ by $50$ repetitions for drug discovery. The black dashed line denotes the target FCR level $10\%$.

Theorems & Definitions (48)

• Proposition 1
• Remark 2.1
• Definition 1
• Theorem 1
• Proposition 2
• Theorem 2
• Corollary 3.1
• Corollary 3.2
• Remark 3.1
• Remark 3.2