A Conformal Prediction Score that is Robust to Label Noise

TL;DR

This work tackles calibrating Conformal Prediction (CP) under noisy validation labels common in medical imaging. It introduces a Noise-Robust CP (NR-CP) score that analytically adjusts the conformal score using the observed noisy label and the noise rate, via the estimator $\hat{S}(x_t,\tilde{y}_t,\epsilon)=(1-\epsilon)S(x_t,\tilde{y}_t)+\epsilon S(x_t)$ with $S(x_t)=(1/k)\sum_i S(x_t,i)$, and forms the test-time prediction set with $\hat{C}_{\epsilon}(x)$. The method demonstrates that NR-CP yields smaller prediction sets than standard Noisy-CP and NRES-CP while preserving the required coverage, and it can be extended to end-to-end scenarios where the noise rate $\epsilon$ is estimated (NR-CP$^*$). Experiments on diverse medical-imaging datasets (e.g., HAM10000, TissueMNIST, PathMNIST, OrganSMNIST) show superior calibration and efficiency, indicating practical applicability for reliable clinical decision support under label noise. The approach also indicates potential generalization to other noise models beyond the uniform flip assumption. Overall, NR-CP provides a principled, scalable way to maintain CP guarantees in the presence of annotation noise, enabling robust uncertainty quantification in medical imaging.

Abstract

Conformal Prediction (CP) quantifies network uncertainty by building a small prediction set with a pre-defined probability that the correct class is within this set. In this study we tackle the problem of CP calibration based on a validation set with noisy labels. We introduce a conformal score that is robust to label noise. The noise-free conformal score is estimated using the noisy labeled data and the noise level. In the test phase the noise-free score is used to form the prediction set. We applied the proposed algorithm to several standard medical imaging classification datasets. We show that our method outperforms current methods by a large margin, in terms of the average size of the prediction set, while maintaining the required coverage.

Figures (1)

• Figure 1: Mean size and coverage as a function of the noise level across different conformal prediction methods for HAM-10000 and $1\!-\!\alpha=0.9$.