Time-uniform conformal and PAC prediction

TL;DR

This work addresses uncertainty quantification for sequential, streaming data by extending distribution-free conformal and PAC prediction to time-uniform guarantees at arbitrary stopping times. It develops split-time-uniform and online time-uniform prediction frameworks (TUC and TUPAC) that yield anytime-valid coverage around a test point, using fixed or dynamically updated transformations and carefully calibrated quantiles. The paper proves equivalence and oracle-width results, demonstrates convergence of prediction-set width to the oracle in suitable settings, and validates the methods with simulations and a real spam-detection dataset, including adaptivity to distribution shifts. These contributions enable memory-efficient, data-driven uncertainty quantification for streaming decision systems, with practical guidance on implementation and extensions to non-IID settings. Collectively, the work provides a rigorous foundation for reliable, sequential conformal and PAC predictions under stopping-time uncertainty.

Abstract

Given that machine learning algorithms are increasingly being deployed to aid in high stakes decision-making, uncertainty quantification methods that wrap around these black box models such as conformal prediction have received much attention in recent years. In sequential settings, where data are observed/generated in a streaming fashion, traditional conformal methods do not provide any guarantee without fixing the sample size. More importantly, traditional conformal methods cannot cope with sequentially updated predictions. As such, we develop an extension of the conformal prediction and related probably approximately correct (PAC) prediction frameworks to sequential settings where the number of data points is not fixed in advance. The resulting prediction sets are anytime-valid in that their expected coverage is at the required level at any time chosen by the analyst even if this choice depends on the data. We present theoretical guarantees for our proposed methods and demonstrate their validity and utility on simulated and real datasets.

Figures (6)

• Figure 1: The true probability content of prediction sets $\{\widehat{C}_{t, \alpha, \delta}\}_{t=1}^{100,000}$ constructed using split conformal, TUC, TUPAC, CS, and SIPI algorithms over 100 replications where $\alpha = \delta = 0.1$. The prediction intervals use the transformation $R(Z) = |Z - \bar{Z}|$ (signed for the SIPI algorithm) where $\bar{Z}$ is the empirical mean based on a separate dataset of 100 standard Normal random variables drawn independently from the data stream. To compute the split TUC and TUPAC prediction sets, we take the function $h$ in Algorithm \ref{['fixed_anytime_alg']} to be the PMF of discretized log Normal random variables with mean 11 and variance 1, respectively (i.e. $h$ is the PMF of $Y = \lfloor X \rfloor$ where $X \sim \text{Lognormal}(11, 1)$). In computing the split CS prediction sets, we choose $u_t$ in Theorem \ref{['fixed_confidence_sequence_pac_thm']} using the Beta-Binomial mixture approach described in howard2022.
• Figure 2: Illustration of dynamic updating of transformation for online TUC and TUPAC prediction.
• Figure 3: The coverage and width of prediction sets when split conformal, CS, TUC, TUPAC, and SIPI based on linear regression are applied to a simulated dataset described in Section \ref{['experiments_sec']} are shown above.
• Figure 4: The true probability content of 90% online TUC prediction intervals $\{\widehat{C}_{t, 0.1}\}_{t=1}^{100,000}$ constructed using Algorithm \ref{['alg:dynamic_algo-illustration']} over 120 replications is shown above.
• Figure 5: The coverage and proportion of noninformative prediction sets when split conformal, CS, TUC, and TUPAC prediction algorithms based on logistic regression, random forest, and a Superlearner ensemble are applied to the problem of spam detection are shown above.

Theorems & Definitions (20)

• Proposition 1
• proof
• Theorem 2.1
• Theorem 2.2
• proof
• Theorem 2.3
• proof
• Theorem 3.1
• Theorem 3.2
• Theorem 4.1