Adaptive conformal classification with noisy labels

TL;DR

The paper addresses conformal prediction under label noise in calibration data by introducing adaptive calibration procedures that account for a contamination process modeled via a linear mixing framework. By characterizing the coverage inflation factor $oldsymbol{ riangle}_k(t)$ and constructing plug-in or CI-based corrections, the authors obtain valid prediction sets that are often more informative than standard conformal sets, across label-conditional and marginal guarantees. The methodology covers known and bounded contamination and includes procedures to estimate the contamination model from data, with theoretical finite-sample guarantees and practical demonstrations on simulations and CIFAR-10H. Overall, this work advances uncertainty quantification in noisy-label regimes, with practical impact for crowdsourced labeling, privacy-preserving data collection, and real-world datasets with imperfect annotations.

Abstract

This paper develops novel conformal prediction methods for classification tasks that can automatically adapt to random label contamination in the calibration sample, leading to more informative prediction sets with stronger coverage guarantees compared to state-of-the-art approaches. This is made possible by a precise characterization of the effective coverage inflation (or deflation) suffered by standard conformal inferences in the presence of label contamination, which is then made actionable through new calibration algorithms. Our solution is flexible and can leverage different modeling assumptions about the label contamination process, while requiring no knowledge of the underlying data distribution or of the inner workings of the machine-learning classifier. The advantages of the proposed methods are demonstrated through extensive simulations and an application to object classification with the CIFAR-10H image data set.

Figures (49)

• Figure 1: Performance of the proposed adaptive conformal prediction methods on CIFAR-10H image data with imperfect labels. The results are shown as a function of the calibration sample size and compared to the performance of standard conformal predictions, which are too conservative due to the presence of inaccurate labels. The dashed line indicates the nominal 90% marginal coverage level.
• Figure 2: Performances of different conformal prediction methods on simulated data with random label contamination of varying strength, as a function of the number of calibration samples. The dashed horizontal line indicates the 90% nominal label-conditional coverage level.
• Figure 3: Performances of different conformal prediction methods on simulated data with contaminated labels from a randomized response model. The results are shown as a function of the known lower bound for the label noise parameter $\epsilon=0.2$ and of the number of possible labels. The number of calibration samples is 10,000. Other details are as in Figure \ref{['fig:exp-synthetic-1-lab-cond-K4-ncal']}.
• Figure 4: Performances of different conformal prediction methods, as a function of the numbers of clean and corrupted samples used to fit the unknown parameter of a randomized response model for the contamination process. The number of possible classes is $K=2$. Other details are as in Figure \ref{['fig:exp-synthetic-1-bounded-ncal-eps0.2-lower']}.
• Figure A1: Visualization of theoretical worst-case bounds for the coverage achieved by standard conformal prediction sets, calibrated for label-conditional coverage, on the CIFAR-10H data with noisy human-assigned labels. The thick black lines correspond to the theoretical bounds, whereas the light gray line denotes the average empirical coverage achieved by the standard conformal prediction approach in our experiments. The dashed line indicates the nominal 90% coverage level.

Theorems & Definitions (55)

• Definition 1: Prediction function
• Theorem 1
• Proposition 1
• Corollary 1
• Theorem 2
• Theorem 3
• Proposition 2
• Theorem 4
• Theorem 5
• Proposition 3