Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Online Conformal Prediction with Efficiency Guarantees

Vaidehi Srinivas

TL;DR

The paper introduces an online conformal prediction framework that targets average miscoverage $(1-\alpha)$ while minimizing the average interval length. It shows a sharp split: for arbitrary sequences, any algorithm achieving a multiplicative volume-approximation $\mu$ must incur at least $\Omega\left( \frac{\log(1/\mathsf{minwidth})}{\log \mu} \alpha T \right)$ mistakes, while for exchangeable sequences there exists a deterministic algorithm with $\mu=1$ achieving near-optimal coverage $ (1-\alpha) - O\left(\sqrt{\frac{\log T}{T}}\right)$ and interval volume bounded by the best fixed interval in hindsight. A unified meta-algorithm underpins both results, but a lower-bound shows no single method can be optimal in both settings, revealing a fundamental separation between arbitrary and exchangeable inputs. The work also develops a uniform-convergence tool for exchangeable sequences to support the exchangeable guarantees and connects the online results to standard conformal prediction. Overall, the paper advances efficiency guarantees for conformal prediction in an online, data-adaptive setting and delineates the limits of unified approaches across data-generating regimes.

Abstract

We study the problem of conformal prediction in a novel online framework that directly optimizes efficiency. In our problem, we are given a target miscoverage rate $α> 0$, and a time horizon $T$. On each day $t \le T$ an algorithm must output an interval $I_t \subseteq [0, 1]$, then a point $y_t \in [0, 1]$ is revealed. The goal of the algorithm is to achieve coverage, that is, $y_t \in I_t$ on (close to) a $(1 - α)$-fraction of days, while maintaining efficiency, that is, minimizing the average volume (length) of the intervals played. This problem is an online analogue to the problem of constructing efficient confidence intervals. We study this problem over arbitrary and exchangeable (random order) input sequences. For exchangeable sequences, we show that it is possible to construct intervals that achieve coverage $(1 - α) - o(1)$, while having length upper bounded by the best fixed interval that achieves coverage in hindsight. For arbitrary sequences however, we show that any algorithm that achieves a $μ$-approximation in average length compared to the best fixed interval achieving coverage in hindsight, must make a multiplicative factor more mistakes than $αT$, where the multiplicative factor depends on $μ$ and the aspect ratio of the problem. Our main algorithmic result is a matching algorithm that can recover all Pareto-optimal settings of $μ$ and number of mistakes. Furthermore, our algorithm is deterministic and therefore robust to an adaptive adversary. This gap between the exchangeable and arbitrary settings is in contrast to the classical online learning problem. In fact, we show that no single algorithm can simultaneously be Pareto-optimal for arbitrary sequences and optimal for exchangeable sequences. On the algorithmic side, we give an algorithm that achieves the near-optimal tradeoff between the two cases.

Online Conformal Prediction with Efficiency Guarantees

TL;DR

The paper introduces an online conformal prediction framework that targets average miscoverage while minimizing the average interval length. It shows a sharp split: for arbitrary sequences, any algorithm achieving a multiplicative volume-approximation must incur at least mistakes, while for exchangeable sequences there exists a deterministic algorithm with achieving near-optimal coverage and interval volume bounded by the best fixed interval in hindsight. A unified meta-algorithm underpins both results, but a lower-bound shows no single method can be optimal in both settings, revealing a fundamental separation between arbitrary and exchangeable inputs. The work also develops a uniform-convergence tool for exchangeable sequences to support the exchangeable guarantees and connects the online results to standard conformal prediction. Overall, the paper advances efficiency guarantees for conformal prediction in an online, data-adaptive setting and delineates the limits of unified approaches across data-generating regimes.

Abstract

We study the problem of conformal prediction in a novel online framework that directly optimizes efficiency. In our problem, we are given a target miscoverage rate , and a time horizon . On each day an algorithm must output an interval , then a point is revealed. The goal of the algorithm is to achieve coverage, that is, on (close to) a -fraction of days, while maintaining efficiency, that is, minimizing the average volume (length) of the intervals played. This problem is an online analogue to the problem of constructing efficient confidence intervals. We study this problem over arbitrary and exchangeable (random order) input sequences. For exchangeable sequences, we show that it is possible to construct intervals that achieve coverage , while having length upper bounded by the best fixed interval that achieves coverage in hindsight. For arbitrary sequences however, we show that any algorithm that achieves a -approximation in average length compared to the best fixed interval achieving coverage in hindsight, must make a multiplicative factor more mistakes than , where the multiplicative factor depends on and the aspect ratio of the problem. Our main algorithmic result is a matching algorithm that can recover all Pareto-optimal settings of and number of mistakes. Furthermore, our algorithm is deterministic and therefore robust to an adaptive adversary. This gap between the exchangeable and arbitrary settings is in contrast to the classical online learning problem. In fact, we show that no single algorithm can simultaneously be Pareto-optimal for arbitrary sequences and optimal for exchangeable sequences. On the algorithmic side, we give an algorithm that achieves the near-optimal tradeoff between the two cases.

Paper Structure

This paper contains 21 sections, 14 theorems, 95 equations, 5 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem statement.
  3. Results
  4. Results for arbitrary sequences.
  5. Results for exchangeable sequences.
  6. "Best of both worlds."
  7. Technical Overview
  8. Contrast to online learning.
  9. Challenges for arbitrary sequences.
  10. Challenges for Exchangeable Sequences.
  11. Related Work
  12. Efficiency guarantees for standard conformal prediction.
  13. Beyond exchangeability.
  14. Online conformal prediction.
  15. "Best of both worlds" for Online Conformal Prediction.
  16. ...and 6 more sections

Key Result

Theorem 1.1

For a given scale lower bound $\mathsf{minwidth} > 0$, multiplicative volume approximation $\mu > 3$, target miscoverage rate $\alpha \ge 0$, and time horizon $T$, we give an algorithm that on any sequence $S$ of length $T$ plays intervals of maximum volume (and therefore average volume) and makes number of mistakes bounded by Moreover, the algorithm is deterministic and therefore robust to an a

Figures (5)

  • Figure 1: $I_t$ achieved more coverage than $I_{\mathrm{current}}^{({t_{\mathrm{prev}}})} = \mu I_{{t_{\mathrm{prev}}}}$ on the sequence so far, so $I_t$ is not contained in $\mu I_{{t_{\mathrm{prev}}}}$. We also know that $I_{{t_{\mathrm{prev}}}}$ and $I_{t}$ must overlap. Thus, $I_t$ must fully contain one of the margins that we add to $I_{{t_{\mathrm{prev}}}}$ to form $\mu I_{{t_{\mathrm{prev}}}}$ (denoted here as the regions between the red dashed lines). This gives a lower bound on the size of $I_t$ in terms of the size of $I_{{t_{\mathrm{prev}}}}$, and therefore a lower bound on the size of $I_{\mathrm{current}}^{(t)}$ in terms of the size of $I_{\mathrm{current}}^{({t_{\mathrm{prev}}})}$
  • Figure 2: We illustrate the lower bound construction in \ref{['thm:arbitrary-order-lower-bound']}. The horizontal axis is the day $t$ of the sequence and the vertical axis is the value of $y_t$. $y_t$ is drawn according to a uniform distribution, and we illustrate this as a box over the support of $y_t$ with depth of color corresponding to the density of the distribution. We divide the input sequence into "phases" of length $\alpha T$. We then construct an input sequence $S$ that has the scale of $y_t$ increasing multiplicatively from one phase to the next (\ref{['subfig:arbitrary-full-sequence']}). We also consider a set of alternate sequences $S_i$, where $S_i$ is the same as $S$ through phase $i$, and then drops to a very small scale for the rest of the sequence (\ref{['subfig:arbitary-alternate-sequence']}). On phase $i$, the algorithm does not know if it is in case (a), where the scale will continue to grow, or case (b), where the sequence will drop off.
  • Figure 3: Symmetric version of the lower bound construction in \ref{['fig:arbitrary-lower-bound']}
  • Figure 4: We illustrate the lower bound construction in \ref{['thm:no-best-of-both-worlds']}. The horizontal axis is the day $t$ of the sequence and the vertical axis is the value of $y_t$. $y_t$ is drawn according to a distribution $\mathcal{D}$ that we illustrate with depth of color corresponding to the density of the distribution. $\mathcal{D}$ has the property that the minimum volume interval that achieves miscoverage $\le \alpha e^i$ over $\mathcal{D}$ is multiplicatively larger than the minimum volume interval that achieves miscoverage $\le \alpha e^{i + 1}$.
  • Figure 5: Symmetric version of i.i.d. lower bound construction in \ref{['fig:iid-lower-bound']}

Theorems & Definitions (25)

  • Theorem 1.1: Informal version of \ref{['cor:optimal-algorithm-for-arbitrary-order']}
  • Theorem 1.2: Informal version of \ref{['thm:arbitrary-order-lower-bound']}
  • Theorem 1.3: Informal version of \ref{['cor:optimal-algorithm-for-exchangeable-sequences']}
  • Theorem 1.4: Informal version of \ref{['thm:no-best-of-both-worlds']}
  • Theorem 2.1: Meta-algorithm Guarantee for Arbitrary Sequences
  • proof
  • Corollary 2.2: Algorithm for Arbitrary Sequences
  • proof
  • Theorem 3.1: Meta-algorithm Guarantee for Exchangeable Sequences
  • proof
  • ...and 15 more