Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online selective conformal inference: adaptive scores, convergence rate and optimality

Pierre Humbert, Ulysse Gazin, Ruth Heller, Etienne Roquain

TL;DR

OnlineSCI extends adaptive conformal inference to online selective tasks, enabling inference only at user-chosen times while guaranteeing non-asymptotic FCP control in adversarial streams and achieving near-optimal instantaneous error rates under iid or autoregressive data. The framework updates adaptive scores and thresholds only on selections, yielding explicit convergence rates to the oracle for both IER and the threshold, with distinct regimes for X-oriented regression settings and informative selection scenarios. Key contributions include new non-asymptotic FCP bounds, convergence-rate theorems for IER and thresholds, optimality results in iid regression and classification contexts, and comprehensive numerical demonstrations. The work lays a foundation for practical online selective inference with adaptive scoring, reporting, and testing capabilities that balance control of errors across selected instances with efficient power in favorable distributional settings.

Abstract

In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of \cite{gibbs2021adaptive}. For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set with an average missed coverage getting close to a pre-specified level $α$ for a long time horizon. We introduce an extended version of this algorithm, called OnlineSCI, allowing the user to additionally select times where such an inference should be made. OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing. While OnlineSCI controls the average missed coverage on the selected in an adversarial setting, our theoretical results also show that it controls the instantaneous error rate (IER) at the selected times, up to a non-asymptotical remainder term. Importantly, our theory covers the case where OnlineSCI updates the point-prediction algorithm at each time step, a property which we refer to as {\it adaptive} capability. We show that the adaptive versions of OnlineSCI can convergence to an optimal solution and provide an explicit convergence rate in each of the aforementioned application cases, under specific mild conditions. Finally, the favorable behavior of OnlineSCI in practice is illustrated by numerical experiments.

Online selective conformal inference: adaptive scores, convergence rate and optimality

TL;DR

OnlineSCI extends adaptive conformal inference to online selective tasks, enabling inference only at user-chosen times while guaranteeing non-asymptotic FCP control in adversarial streams and achieving near-optimal instantaneous error rates under iid or autoregressive data. The framework updates adaptive scores and thresholds only on selections, yielding explicit convergence rates to the oracle for both IER and the threshold, with distinct regimes for X-oriented regression settings and informative selection scenarios. Key contributions include new non-asymptotic FCP bounds, convergence-rate theorems for IER and thresholds, optimality results in iid regression and classification contexts, and comprehensive numerical demonstrations. The work lays a foundation for practical online selective inference with adaptive scoring, reporting, and testing capabilities that balance control of errors across selected instances with efficient power in favorable distributional settings.

Abstract

In a supervised online setting, quantifying uncertainty has been proposed in the seminal work of \cite{gibbs2021adaptive}. For any given point-prediction algorithm, their method (ACI) produces a conformal prediction set with an average missed coverage getting close to a pre-specified level for a long time horizon. We introduce an extended version of this algorithm, called OnlineSCI, allowing the user to additionally select times where such an inference should be made. OnlineSCI encompasses several prominent online selective tasks, such as building prediction intervals for extreme outcomes, classification with abstention, and online testing. While OnlineSCI controls the average missed coverage on the selected in an adversarial setting, our theoretical results also show that it controls the instantaneous error rate (IER) at the selected times, up to a non-asymptotical remainder term. Importantly, our theory covers the case where OnlineSCI updates the point-prediction algorithm at each time step, a property which we refer to as {\it adaptive} capability. We show that the adaptive versions of OnlineSCI can convergence to an optimal solution and provide an explicit convergence rate in each of the aforementioned application cases, under specific mild conditions. Finally, the favorable behavior of OnlineSCI in practice is illustrated by numerical experiments.

Paper Structure

This paper contains 66 sections, 39 theorems, 246 equations, 9 figures, 4 algorithms.

Table of Contents

  1. Introduction
  2. Online conformal testing
  3. Related work and terminology
  4. Preliminaries
  5. Setting
  6. Leading examples
  7. Error rates
  8. OnlineSCI
  9. FCP bound
  10. Convergence rates
  11. Benchmark IER function
  12. Main results
  13. Results for $X$-oriented selections and regression model
  14. Results for iid regression model
  15. Results for auto-regressive model
  16. ...and 51 more sections

Key Result

Theorem 2.1

For any sequence $(X_{t},Y_{t})_{t\geq 1}$, the procedure OnlineSCI is such that for all $t\geq 1$, where recall that $J(t)=\sum_{k=1}^{t-1} S_{k}(X_{k}, q_{k}^{{\hbox{\tiny SCI}}}) + 1$, see def:Jt. In particular, further assuming that the sequence of $(\gamma_j)_{j\geq 1}$ is nonincreasing, we have for all $t\geq 1$,

Figures (9)

  • Figure 1: False coverage proportion on the selection (FCP) along time for two methods: ACI with decaying step sizes angelopoulos2024online (red) versus the new proposed procedure OnlineSCI (blue). Left panel: online selective classification (FCP is mis-classification rate, selection rule is prediction sets of size one, see § \ref{['subsec-simClassification']} and an illustration in Figure \ref{['fig:intro-selective-classif-combined_figure']} for more details). Right panel: online selective prediction intervals (selection if previous outcome is below a threshold, see § \ref{['sec:elec']} and an illustration in Figure \ref{['fig:EDF']} for more details).
  • Figure 2: Illustration of the online conformal testing application for Cifar10. Classes 'plane' and 'truck' correspond to $Y=0$ (null) and 'cat' to $Y=1$ (alternative). A red square indicates a false discovery and a green square a true discovery. No color means no discovery. LORD++ ramdas2017online and Feedback LORD lu2025feedback do not make any discovery.
  • Figure 3: Online conformal testing: results for OnlineSCI and the oracle policy on a particular data set. Top left panel: $\mathrm{FDP}_t$ versus time step $t$. Top right panel: method threshold $q_t$ versus time step $t$. Bottom left and right panel: $\mathrm{IER}_t$ and instantaneous power at each time step. Note that LORD++ is not reported because it does not make any discovery.
  • Figure 4: Online selective classification: results for OnlineSCI and the oracle policy on a particular data set. Top left panel: $\mathrm{FCP}_t$ versus time step $t$. Top right panel: method threshold $q_t$ versus time step $t$. Bottom left and right panel: $\mathrm{IER}_t$ and instantaneous power at each time step.
  • Figure 5: Online conformal testing with adaptive OnlineSCI: same as Figure \ref{['fig:ND_xp']} top panels with $5$ different data set generations from the same model.
  • ...and 4 more figures

Theorems & Definitions (62)

  • Definition 1
  • Remark 2.1
  • Definition 2
  • Remark 2.2
  • Theorem 2.1
  • Remark 2.3
  • Theorem 3.1
  • Theorem 3.2
  • Remark 3.1: Typical rate under concentration
  • Remark 3.2: Choosing $\beta$
  • ...and 52 more