Relevance-Aware Thresholding in Online Conformal Prediction for Time Series

TL;DR

This work addresses the problem that online conformal prediction (OCP) for time series often relies on binary inside/outside errors, which can produce overly wide intervals under distribution shift. It introduces a relevance-aware thresholding framework by defining a family of functions $f_{comega,v,\mu_t}$ that quantify how far the ground truth is from the predicted interval via the distance $s_t - q_t$ and feed this into updates of the next threshold $q_{t+1}$ for state-of-the-art OCP methods (PID and ECI). The authors provide a formal construction with scale-adaptive sigmoid components, prove long-run coverage properties under modified updates, and demonstrate that the modified PID and ECI methods yield tighter prediction intervals on real time-series data (Amazon, Google, Microsoft stock prices, and Delhi temperature) while maintaining validity. This relevance-aware approach improves both robustness to distribution shifts and decision-making usefulness in practical settings.

Abstract

Uncertainty quantification has received considerable interest in recent works in Machine Learning. In particular, Conformal Prediction (CP) gains ground in this field. For the case of time series, Online Conformal Prediction (OCP) becomes an option to address the problem of data distribution shift over time. Indeed, the idea of OCP is to update a threshold of some quantity (whether the miscoverage level or the quantile) based on the distribution observation. To evaluate the performance of OCP methods, two key aspects are typically considered: the coverage validity and the prediction interval width minimization. Recently, new OCP methods have emerged, offering long-run coverage guarantees and producing more informative intervals. However, during the threshold update step, most of these methods focus solely on the validity of the prediction intervals~--~that is, whether the ground truth falls inside or outside the interval~--~without accounting for their relevance. In this paper, we aim to leverage this overlooked aspect. Specifically, we propose enhancing the threshold update step by replacing the binary evaluation (inside/outside) with a broader class of functions that quantify the relevance of the prediction interval using the ground truth. This approach helps prevent abrupt threshold changes, potentially resulting in narrower prediction intervals. Indeed, experimental results on real-world datasets suggest that these functions can produce tighter intervals compared to existing OCP methods while maintaining coverage validity.

Figures (6)

• Figure 1: $f_{\omega,v,\mu_t}$ with $\alpha=0.1,\omega=1, v=4 \;(l=1)$ and $\mu_t=20$.
• Figure 2: Plots of $f_{\omega,v,\mu_t}$ with $\alpha=0.1, \omega=1 \;(l=1)$ and $\mu_t=10$.
• Figure 3: Plots of $f_{\omega,v,\mu_t}$ with $\alpha=0.1, v_1=1, v_2=10$ and $\mu_t=10$.
• Figure 4: PID vs modified PID performances on the Amazon stock price dataset. The plots show Theta as the regressor.
• Figure 5: ECI vs modified ECI on the Microsoft stock price dataset (AR regressor).

Theorems & Definitions (6)

• theorem thmcountertheorem
• theorem thmcountertheorem
• theorem thmcountertheorem
• proof : Theorem \ref{['theo: PID_bis']}
• proof : Theorem \ref{['theo: PID_others']}
• proof : Theorem \ref{['theo:ECI']}