Questioning the Coverage-Length Metric in Conformal Prediction: When Shorter Intervals Are Not Better

TL;DR

The paper questions the sufficiency of the standard coverage-length metrics used to evaluate conformal prediction (CP) by introducing the Prejudicial Trick (PT), a randomized procedure that can reduce average interval length while maintaining marginal coverage. PT achieves this by returning a null interval with probability $1-p$ and a CP interval with adjusted miscoverage $\alpha' = 1-\frac{1-\alpha}{p}$ with probability $p$, thereby preserving $\mathbb{P}(Y\in\mathcal{C}_{1-\alpha}(X)) \ge 1-\alpha$ but introducing instability across runs and unfairness across subgroups. The authors establish conditions under which PT preserves marginal and conditional coverage and, under misspecification or differentiable-length assumptions, can shorten expected interval length; they also provide empirical evidence across regression and classification tasks. To mitigate the risk of such vacuous improvements, they introduce Interval Stability (IS), a metric that quantifies the variability of the returned interval across runs and data, serving as a diagnostic tool alongside traditional coverage and length metrics. Overall, the work cautions that current evaluation criteria may be incomplete and offers IS as a practical safeguard to detect spurious randomness in CP methods.

Abstract

Conformal prediction (CP) has become a cornerstone of distribution-free uncertainty quantification, conventionally evaluated by its coverage and interval length. This work critically examines the sufficiency of these standard metrics. We demonstrate that the interval length might be deceptively improved through a counter-intuitive approach termed Prejudicial Trick (PT), while the coverage remains valid. Specifically, for any given test sample, PT probabilistically returns an interval, which is either null or constructed using an adjusted confidence level, thereby preserving marginal coverage. While PT potentially yields a deceptively lower interval length, it introduces practical vulnerabilities: the same input can yield completely different prediction intervals across repeated runs of the algorithm. We formally derive the conditions under which PT achieves these misleading improvements and provides extensive empirical evidence across various regression and classification tasks. Furthermore, we introduce a new metric interval stability which helps detect whether a new CP method implicitly improves the length based on such PT-like techniques.

Figures (15)

• Figure 1: Comparing the (a) marginal coverage and (b, c, d) conditional coverage between VCP with and without PT. Results demonstrate that PT would not significantly change the marginal coverage; and PT has better conditional coverage compared to the base algorithm.
• Figure 2: Illustration of coverage and interval length.
• Figure 3: Illustration of Example \ref{['example: PatientRecovery']}. Doctor Alice and Bob both achieve $60\%$ accuracy. Bob is more precise regarding length, but the corresponding strategy is not practically valid.
• Figure 4: The illustration of Prejudicial Trick (PT). To obtain a $1-\alpha$ confidence interval, PT first assigns empty sets for a $1-p$ subset of the test points, and assigns $1-\alpha^\prime$ confidence interval for the remaining test points where $\alpha^\prime < \alpha$. The returned confidence interval still satisfies $\mathbb{P}(Y \in \mathcal{C}(X)) \geq 1-\alpha$ by setting a proper $\alpha^\prime$.
• Figure 5: PT fails to improve length when the conditions on the distribution of non-conformity score are not safisfied.

Theorems & Definitions (27)

• Example 1: The Pitfalls of Length.
• Theorem 1: Theorem Summary
• Remark 2: PT Hacks the Coverage-Length Metric
• Remark 3: Practical Relevance of the PT with Current CP Methods
• Remark 4: Why Study PT In Conformal Prediction
• Example 2: Synthetic Dataset
• Remark 5: The extension of PT
• Proposition 1: Relation Between PT And Localized CP
• Theorem 6: Marginal Coverage Guarantee
• Theorem 7: Conditional Coverage Guarantee