Conformal Prediction for Deep Classifier via Label Ranking
Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, Hongxin Wei
TL;DR
This work addresses the inefficiency of conformal prediction in deep multiclass classifiers caused by long-tailed softmax probabilities. It introduces Sorted Adaptive Prediction Sets (SAPS), which discard all probability values except the maximum softmax probability and rely on label ranking to construct prediction sets with guaranteed marginal coverage. The authors provide theoretical insights showing probability magnitudes are largely unnecessary for CP and demonstrate that SAPS achieves dramatically smaller sets and higher conditional coverage than APS and RAPS across ImageNet and CIFAR benchmarks, including under distribution shifts. The method is simple to implement on top of any pretrained classifier and improves instance-wise uncertainty communication, with practical implications for risk-sensitive applications.
Abstract
Conformal prediction is a statistical framework that generates prediction sets containing ground-truth labels with a desired coverage guarantee. The predicted probabilities produced by machine learning models are generally miscalibrated, leading to large prediction sets in conformal prediction. To address this issue, we propose a novel algorithm named $\textit{Sorted Adaptive Prediction Sets}$ (SAPS), which discards all the probability values except for the maximum softmax probability. The key idea behind SAPS is to minimize the dependence of the non-conformity score on the probability values while retaining the uncertainty information. In this manner, SAPS can produce compact prediction sets and communicate instance-wise uncertainty. Extensive experiments validate that SAPS not only lessens the prediction sets but also broadly enhances the conditional coverage rate of prediction sets.