Adaptive Conformal Inference by Betting

TL;DR

The paper tackles adaptive conformal inference for streaming, non-exchangeable data by proposing a parameter-free, betting-based approach that uses coin betting strategies to update prediction set radii without tuning. By reframing miscoverage as online quantile learning of $S_t=|Y_t-\hat f_t(X_t)|$ with the pinball loss at $\beta=1-\alpha$, it derives KT-based and Blackbox ON-S algorithms that achieve sublinear regret and guarantee long-run coverage $\mathbb{P}(Y_t\in\hat C_t(s_t))\approx 1-\alpha$. A key theoretical result shows that, under bounded nonconformity scores $S_i\in[0,D]$, the miscoverage converges to the nominal level: $\lim_{t\to\infty} \left| \frac{1}{t}\sum_{i=1}^t \mathbf{1}\{Y_i \notin \hat C_i(s_i)\} - \alpha \right| = 0$. Empirically, the method matches or closely rivals tuned baselines in changepoint and time-series experiments, while avoiding parameter tuning and offering robust adaptivity to distribution shifts and multi-step forecasts.

Abstract

Conformal prediction is a valuable tool for quantifying predictive uncertainty of machine learning models. However, its applicability relies on the assumption of data exchangeability, a condition which is often not met in real-world scenarios. In this paper, we consider the problem of adaptive conformal inference without any assumptions about the data generating process. Existing approaches for adaptive conformal inference are based on optimizing the pinball loss using variants of online gradient descent. A notable shortcoming of such approaches is in their explicit dependence on and sensitivity to the choice of the learning rates. In this paper, we propose a different approach for adaptive conformal inference that leverages parameter-free online convex optimization techniques. We prove that our method controls long-term miscoverage frequency at a nominal level and demonstrate its convincing empirical performance without any need of performing cumbersome parameter tuning.

Figures (6)

• Figure 1: Comparison of the proposed conformal predictor against those trained via OGD/SF-OGD with different learning rates.
• Figure 2: Comparison of the proposed conformal predictor against a couple of competitors. The performance of the competitors is sensitive to the choice of the learning rate. We do not plot results observed for the first 25 point. Results are smoothed using rolling window of size 10.
• Figure 3: Comparison of the proposed conformal predictor against a couple of competitors on time series data. (add a similar plot with changing learning rates and summary stats; make a point about looseness in the end, and fast convergence to (slightly sub-optimal) solution in a handful of iterations)
• Figure 4: Comparison of the proposed conformal predictor against a couple of competitors on time series data (5-step ahead forecasting). Left row: the results zoomed into the first step, right row: the results zoomed into the last step.
• Figure 5: Performance of several methods when a linear model, whose coefficients are learned using weighted least squares.

Theorems & Definitions (3)

• theorem 1
• theorem 1
• proof