Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Relationship between No-Regret Learning and Online Conformal Prediction

Ramya Ramalingam, Shayan Kiyani, Aaron Roth

TL;DR

The paper addresses obtaining reliable prediction sets in online conformal prediction under adversarial and group-structured settings. It connects no-regret learning to coverage: external regret suffices for marginal coverage only in IID, while swap regret aligns with threshold-calibrated and multivalid coverage, extending to group-conditioned guarantees. It then shows that follow-the-regularized-leader (FTRL) algorithms, including online gradient descent, can provide group-conditional coverage guarantees, and introduces Group Conditional ACI (GCACI) as a practical instantiation with provable bounds. Empirical results on time-series, synthetic shift, and real distribution-shift data demonstrate GCACI's faster convergence to target group coverage compared to existing methods, with robust performance across varying group structures.

Abstract

Existing algorithms for online conformal prediction -- guaranteeing marginal coverage in adversarial settings -- are variants of online gradient descent (OGD), but their analyses of worst-case coverage do not follow from the regret guarantee of OGD. What is the relationship between no-regret learning and online conformal prediction? We observe that although standard regret guarantees imply marginal coverage in i.i.d. settings, this connection fails as soon as we either move to adversarial environments or ask for group conditional coverage. On the other hand, we show a tight connection between threshold calibrated coverage and swap-regret in adversarial settings, which extends to group-conditional (multi-valid) coverage. We also show that algorithms in the follow the perturbed leader family of no regret learning algorithms (which includes online gradient descent) can be used to give group-conditional coverage guarantees in adversarial settings for arbitrary grouping functions. Via this connection we analyze and conduct experiments using a multi-group generalization of the ACI algorithm of Gibbs & Candes [2021] (arXiv:2106.00170).

The Relationship between No-Regret Learning and Online Conformal Prediction

TL;DR

The paper addresses obtaining reliable prediction sets in online conformal prediction under adversarial and group-structured settings. It connects no-regret learning to coverage: external regret suffices for marginal coverage only in IID, while swap regret aligns with threshold-calibrated and multivalid coverage, extending to group-conditioned guarantees. It then shows that follow-the-regularized-leader (FTRL) algorithms, including online gradient descent, can provide group-conditional coverage guarantees, and introduces Group Conditional ACI (GCACI) as a practical instantiation with provable bounds. Empirical results on time-series, synthetic shift, and real distribution-shift data demonstrate GCACI's faster convergence to target group coverage compared to existing methods, with robust performance across varying group structures.

Abstract

Existing algorithms for online conformal prediction -- guaranteeing marginal coverage in adversarial settings -- are variants of online gradient descent (OGD), but their analyses of worst-case coverage do not follow from the regret guarantee of OGD. What is the relationship between no-regret learning and online conformal prediction? We observe that although standard regret guarantees imply marginal coverage in i.i.d. settings, this connection fails as soon as we either move to adversarial environments or ask for group conditional coverage. On the other hand, we show a tight connection between threshold calibrated coverage and swap-regret in adversarial settings, which extends to group-conditional (multi-valid) coverage. We also show that algorithms in the follow the perturbed leader family of no regret learning algorithms (which includes online gradient descent) can be used to give group-conditional coverage guarantees in adversarial settings for arbitrary grouping functions. Via this connection we analyze and conduct experiments using a multi-group generalization of the ACI algorithm of Gibbs & Candes [2021] (arXiv:2106.00170).

Paper Structure

This paper contains 19 sections, 18 theorems, 56 equations, 3 figures, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our Results
  3. Regret and Coverage
  4. External Regret
  5. Swap Regret
  6. Coverage Guarantees Beyond Regret
  7. Additional Related Work
  8. Definitions
  9. Coverage
  10. Regret
  11. Coverage Guarantees Through Regret
  12. Coverage Guarantees for FTRL Algorithms
  13. Group Conditional ACI
  14. Experiments
  15. Time Series Data
  16. ...and 4 more sections

Key Result

Lemma 3.1

Fix a distribution $\mathcal{D}$, and let $\tau^*$ be the $q$-th quantile of $\mathcal{D}$. Then, assuming $\mathcal{D}$ is an $(\alpha,\rho, r)$-smooth distribution, for any other threshold $\tau'$,

Figures (3)

  • Figure 1: Comparison of convergence rates between GroupACI (GCACI) and MVP for group coverage. Each curve captures the averaged miscoverage over time of a single group. The top left is the comparison for time series data, the top right is UCI Airfoil data, and the bottom is Folktables data. Note that group size varies for curves in each graph.
  • Figure 2: $||\theta_t||_{\infty}$ over time for all three experiments, when running GCACI.
  • Figure 3: Convergence rate of GCACI as a function of the learning rate $\eta$. Each plot measures (across different chosen learning rates) the earliest time-step at which coverage for each group is within $\epsilon = 0.01$ of the desired coverage for the rest of the transcript.

Theorems & Definitions (40)

  • Definition 2.1: Transcript
  • Definition 2.2: Coverage, Coverage Error
  • Definition 2.3: Group
  • Definition 2.4: Group conditional Coverage, Group Size
  • Definition 2.5: Threshold-calibrated coverage
  • Definition 2.6: Multivalid coverage
  • Definition 2.7: $\Phi$-regret phiregret
  • Definition 2.8: $\Phi$-group conditional regret
  • Definition 2.9: $(\alpha, \rho,r)$-smoothness
  • Lemma 3.1
  • ...and 30 more