On some practical challenges of conformal prediction
Liang Hong, Noura Raydan Nasreddine
TL;DR
The paper investigates practical challenges of conformal prediction in regression, notably the exact determination of prediction regions, computational cost, and region shape. It demonstrates that monotonicity of the non-conformity measure $M$ does not guarantee the key ordering property or interval-shaped regions, and that no universal rule suffices for choosing $M$. To address these issues, the authors propose a simple, parsimious non-conformity measure with a two-parameter quadratic form that can produce exact, interval conformal regions (left-bounded, right-bounded, or bounded) and retains finite-sample validity; they illustrate the approach with simulations showing nominal coverage and favorable efficiency, especially when the model is well-specified. The Appendix extends the ideas to unsupervised conformal prediction, offering analogous results and an additional strategy using a different polynomial form. Overall, the work provides a practical, easy-to-implement route to address common conformal-prediction challenges and clarifies the limitations of relying on monotonicity alone.
Abstract
Conformal prediction is a model-free machine learning method for creating prediction regions with a guaranteed coverage probability level. However, a data scientist often faces three challenges in practice: (i) the determination of a conformal prediction region is only approximate, jeopardizing the finite-sample validity of prediction, (ii) the computation required could be prohibitively expensive, and (iii) the shape of a conformal prediction region is hard to control. This article offers new insights into the relationship among the monotonicity of the non-conformity measure, the monotonicity of the plausibility function, and the exact determination of a conformal prediction region. Based on these new insights, we propose a simple strategy to alleviate the three challenges simultaneously.