Noise-Adaptive Conformal Classification with Marginal Coverage

TL;DR

The paper tackles conformal classification under label noise by introducing an adaptive calibration that preserves marginal coverage despite deviations from exchangeability. It derives a marginal-coverage inflation factor and develops both finite-sample ($\delta^{\mathrm{FS}}(n)$) and asymptotic ($\delta^{\mathrm{asy}}(n)$) corrections, enabling efficient, informative prediction sets. The approach leverages an estimated contamination model $T$ (and its inverse $W$) to adjust thresholds and guarantee coverage; it offers rigorous finite-sample bounds and practical asymptotic approximations based on a Generalized Brownian Bridge. Empirical results on synthetic data and real datasets CIFAR-10H and BigEarthNet demonstrate substantially improved informativeness while maintaining the desired marginal coverage, with the asymptotic method particularly robust when contamination deviates from simple models. The work provides a versatile framework for reliable uncertainty quantification in noisy-label scenarios and suggests directions for extending to regression and uncertainty in $T$ estimation.

Abstract

Conformal inference provides a rigorous statistical framework for uncertainty quantification in machine learning, enabling well-calibrated prediction sets with precise coverage guarantees for any classification model. However, its reliance on the idealized assumption of perfect data exchangeability limits its effectiveness in the presence of real-world complications, such as low-quality labels -- a widespread issue in modern large-scale data sets. This work tackles this open problem by introducing an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise, leading to informative prediction sets with tight marginal coverage guarantees even in those challenging scenarios. We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets, including CIFAR-10H and BigEarthNet.

Figures (23)

• Figure 1: Performances of different conformal methods on simulated data with labels contaminated by a two-level randomized response model, as function of the number of calibration samples. The empirical coverage and average size of the prediction sets are stratified based on the strength $\epsilon$ of the label contamination process. The number of classes is $K=4$.
• Figure 2: Performances of different conformal methods on simulated data with labels contaminated by a two-level randomized response model, as function of the number of calibration samples. The reported empirical coverage and average size of the prediction sets are stratified based on the contamination model parameter $\nu$. Other details are as in Figure \ref{['fig:exp1_synthetic1_ntrain10000_K4_nu0.2_marginal_RRB_optimisticFALSE']}.
• Figure 3: Performances of conformal prediction methods on simulated data with contaminated labels. The contamination process is the two-level randomized response model with $\epsilon=0.2$ and $\nu = 0.8$. The reported empirical coverage and average size of the prediction sets are stratified based on the number of possible labels $K$. Other details are as in Figure \ref{['fig:exp1_synthetic1_ntrain10000_K4_nu0.2_marginal_RRB_optimisticFALSE']}.
• Figure 4: Performances of conformal prediction methods on simulated data with contaminated labels. The label contamination process is the two-level randomized response model with $\epsilon=0.05$ and $\nu = 0.2$. The vertical axis for Size is truncated to highlight differences between the Standard and marginal Adaptive+ methods, which would otherwise be obscured by the full scale required to accommodate the Adaptive+ (label-cond) method, especially for small samples. Other details are as in Figure \ref{['fig:exp6_synthetic1_ntrain10000_eps0.200000_nu0.8_marginal_RRB_optimisticTRUE']}.
• Figure 5: Performances of the conformal prediction methods on the CIFAR-10H data set with contaminated labels. The results are shown as a function of the number of calibration samples. The dashed line indicates the nominal 90% marginal coverage level.

Theorems & Definitions (26)

• Definition 1: Prediction function
• Theorem 1
• Theorem 2
• Theorem 3
• Theorem 4
• Theorem 5
• Theorem 6
• Proposition A1: e.g., from lei2013distribution or romano2020classification
• Corollary A1
• Proposition A2