Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online conformal prediction with decaying step sizes

Anastasios N. Angelopoulos, Rina Foygel Barber, Stephen Bates

TL;DR

This work tackles online uncertainty quantification by designing prediction sets with decaying step sizes that provide simultaneous guarantees in both adversarial and IID settings. The method updates a quantile threshold via q_{t+1} = q_t + \eta_t (\mathbb{1}_{Y_t \notin \mathcal{C}_t(X_t)} - \alpha), yielding long-run coverage at level $1-\alpha$ under decaying steps and convergence of the threshold to the optimal quantile in IID data. Theoretical results show robust long-run bounds for arbitrary score sequences and step sizes, while IID results prove convergence of the instantaneous coverage and threshold under suitable decay, with rate $\mathcal{O}(T^{-1/2-\varepsilon})$ for step sizes $\eta_t \propto t^{-1/2-\varepsilon}$. Empirically, decaying-step online conformal prediction yields more stable, near-nominal coverage across datasets (Elec2, Imagenet, M4) and avoids the overreaction seen with fixed-step methods, demonstrating practical viability and robustness to distribution shifts. Overall, the work unifies online conformal prediction with online convex optimization, offering a principled way to balance worst-case validity and population-quantile estimation in sequential settings.

Abstract

We introduce a method for online conformal prediction with decaying step sizes. Like previous methods, ours possesses a retrospective guarantee of coverage for arbitrary sequences. However, unlike previous methods, we can simultaneously estimate a population quantile when it exists. Our theory and experiments indicate substantially improved practical properties: in particular, when the distribution is stable, the coverage is close to the desired level for every time point, not just on average over the observed sequence.

Online conformal prediction with decaying step sizes

TL;DR

This work tackles online uncertainty quantification by designing prediction sets with decaying step sizes that provide simultaneous guarantees in both adversarial and IID settings. The method updates a quantile threshold via q_{t+1} = q_t + \eta_t (\mathbb{1}_{Y_t \notin \mathcal{C}_t(X_t)} - \alpha), yielding long-run coverage at level under decaying steps and convergence of the threshold to the optimal quantile in IID data. Theoretical results show robust long-run bounds for arbitrary score sequences and step sizes, while IID results prove convergence of the instantaneous coverage and threshold under suitable decay, with rate for step sizes . Empirically, decaying-step online conformal prediction yields more stable, near-nominal coverage across datasets (Elec2, Imagenet, M4) and avoids the overreaction seen with fixed-step methods, demonstrating practical viability and robustness to distribution shifts. Overall, the work unifies online conformal prediction with online convex optimization, offering a principled way to balance worst-case validity and population-quantile estimation in sequential settings.

Abstract

We introduce a method for online conformal prediction with decaying step sizes. Like previous methods, ours possesses a retrospective guarantee of coverage for arbitrary sequences. However, unlike previous methods, we can simultaneously estimate a population quantile when it exists. Our theory and experiments indicate substantially improved practical properties: in particular, when the distribution is stable, the coverage is close to the desired level for every time point, not just on average over the observed sequence.
Paper Structure (34 sections, 10 theorems, 53 equations, 3 figures, 1 table)

This paper contains 34 sections, 10 theorems, 53 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Method and Setup
  3. Related work
  4. Our contribution
  5. Main results in the adversarial setting
  6. Decreasing step sizes
  7. Arbitrary step sizes
  8. Results for I.I.D. data
  9. Results with a pretrained score function
  10. Results with online training of the score function
  11. Experiments
  12. Results
  13. The thresholds $q_t$.
  14. The instantaneous coverage $\mathsf{Coverage}_t(q_t)$.
  15. Long-run coverage and rolling coverage.
  16. ...and 19 more sections

Key Result

Theorem 1

Let $(X_1,Y_1),(X_2,Y_2),\dots$ be an arbitrary sequence of data points, and let $s_t:\mathcal{X}\times\mathcal{Y}\rightarrow[0,B]$ be arbitrary functions. Let $\eta_t$ be a positive and nonincreasing sequence of step sizes, and fix an initial threshold $q_1\in[0,B]$. Then online conformal predictio for all $T\geq 1$.

Figures (3)

  • Figure 1: Elec2 results. From left to right, the panels display the following (over all times $t$): first, the value of the threshold $q_t$; second, the instantaneous coverage $\mathsf{Coverage}_t(q_t)$; third, the long-run coverage $\frac{1}{t}\sum_{r=1}^t {\mathbbm{1}}_{{Y_r\in\mathcal{C}_r(X_r)}}$; and fourth, the rolling coverage, averaged over a rolling window of 1000 time points.
  • Figure 3: Density plots of results on M4 datasets. These plots show the same quantities as in Table \ref{['tab:M4']}, but now as histograms over the time-series in M4.
  • Figure 4: Simulation comparison of decaying step size and 'decay+adapt'. The raw score sequence is shown in blue, the decaying step size sequence is in orange, and 'decay+adapt' is in green.

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Proposition 1
  • Theorem 3
  • Corollary 1
  • Proposition 2
  • Proposition 3
  • Theorem 4
  • Corollary 2
  • Lemma 1