Length Optimization in Conformal Prediction
Shayan Kiyani, George Pappas, Hamed Hassani
TL;DR
This work addresses the tension between conditional validity and length efficiency in conformal prediction by introducing Conformal Prediction with Length-Optimization (CPL). The authors formulate a minimax duality framework that characterizes optimal length via level-set interpretations of conditional densities and propose a practical finite-sample algorithm that optimizes adaptive thresholds over a covariate shift class $\mathcal{F}$ and a structured prediction class $\mathcal{H}$. They establish strong duality in the infinite-sample setting and finite-sample guarantees under realizability or bounded complexity, then demonstrate substantial length reductions across marginal, group-conditional, and covariate-shift scenarios on regression, text, and vision tasks, with open-source code available. The results indicate CPL can provide tighter, conditionally valid prediction sets across diverse data regimes, offering a scalable and principled route to more informative uncertainty quantification.
Abstract
Conditional validity and length efficiency are two crucial aspects of conformal prediction (CP). Conditional validity ensures accurate uncertainty quantification for data subpopulations, while proper length efficiency ensures that the prediction sets remain informative. Despite significant efforts to address each of these issues individually, a principled framework that reconciles these two objectives has been missing in the CP literature. In this paper, we develop Conformal Prediction with Length-Optimization (CPL) - a novel and practical framework that constructs prediction sets with (near-) optimal length while ensuring conditional validity under various classes of covariate shifts, including the key cases of marginal and group-conditional coverage. In the infinite sample regime, we provide strong duality results which indicate that CPL achieves conditional validity and length optimality. In the finite sample regime, we show that CPL constructs conditionally valid prediction sets. Our extensive empirical evaluations demonstrate the superior prediction set size performance of CPL compared to state-of-the-art methods across diverse real-world and synthetic datasets in classification, regression, and large language model-based multiple choice question answering. An Implementation of our algorithm can be accessed at the following link: https://github.com/shayankiyani98/CP.