Exploring the Noise Robustness of Online Conformal Prediction

TL;DR

This work tackles uncertainty quantification under online distribution shifts by focusing on online conformal prediction under uniform label noise. It identifies a persistent coverage gap caused by noisy labels and introduces Noise Robust Online Conformal Prediction (NR-OCP) which uses a robust pinball loss to update the conformal threshold without ground-truth labels. The authors prove that the robust loss yields unbiased estimates of the clean pinball loss in expectation and establish convergence rates of $\mathcal{O}(T^{-1/2})$ for empirical and expected coverage errors under both constant and dynamic learning rates. Empirically, NR-OCP substantially reduces the coverage gap while maintaining precise long-run coverage and improved set efficiency on CIFAR-100 and ImageNet across multiple non-conformity scores and architectures. The method is simple to implement and complements existing online conformal prediction frameworks, with potential applicability to real-world noisy data streams.

Abstract

Conformal prediction is an emerging technique for uncertainty quantification that constructs prediction sets guaranteed to contain the true label with a predefined probability. Recent work develops online conformal prediction methods that adaptively construct prediction sets to accommodate distribution shifts. However, existing algorithms typically assume perfect label accuracy which rarely holds in practice. In this work, we investigate the robustness of online conformal prediction under uniform label noise with a known noise rate, in both constant and dynamic learning rate schedules. We show that label noise causes a persistent gap between the actual mis-coverage rate and the desired rate $α$, leading to either overestimated or underestimated coverage guarantees. To address this issue, we propose Noise Robust Online Conformal Prediction (dubbed NR-OCP) by updating the threshold with a novel robust pinball loss, which provides an unbiased estimate of clean pinball loss without requiring ground-truth labels. Our theoretical analysis shows that NR-OCP eliminates the coverage gap in both constant and dynamic learning rate schedules, achieving a convergence rate of $\mathcal{O}(T^{-1/2})$ for both empirical and expected coverage errors under uniform label noise. Extensive experiments demonstrate the effectiveness of our method by achieving both precise coverage and improved efficiency.

Figures (3)

• Figure 1: Performance of ACI gibbs2021adaptive with a constant learning rate $\eta=0.05$ under different noise rates, using ResNet18 on CIFAR-100 and ImageNet datasets. The results validate that label noise introduces a coverage gap, with higher noise rates resulting in a more pronounced gap.
• Figure 2: Long-run coverage and local coverage performance of various methods under uniform noisy labels with noise rate $\epsilon=0.05$. We apply robust pinball loss to ACI with (a) constant gibbs2021adaptive and (b) dynamic learning rate DBLP:conf/icml/AngelopoulosBB24, and (c) SAOCP bhatnagar2023improved. We employ LAC to generate prediction sets with $\alpha=0.1$, using ResNet18 on CIFAR100. "Baseline" and "Clean" denote the online conformal prediction with standard pinball loss, using noisy and clean labels.
• Figure 3: Performance of standard online conformal prediction under different noise rates, with a ResNet18 model on CIFAR-10 and CIFAR-100 datasets. We use noisy labels to update the threshold with decaying learning rates $\eta_t=1/t^{1/2+\varepsilon}$ where $\varepsilon=0.1$.

Theorems & Definitions (41)

• Proposition 3.1
• Proposition 4.1
• Proposition 4.2
• Proposition 4.3
• Remark 4.4
• Proposition 4.5
• Proposition 4.6
• Proposition B.1
• Proposition B.3
• Lemma E.1