Selecting informative conformal prediction sets with false coverage rate control

TL;DR

The paper tackles constructing conformal prediction sets after a data-driven informativeness selection, with finite-sample control of the false coverage rate on the selected subset. It introduces two methods, InfoSP and InfoSCOP, that fuse informative selection with conformal prediction, ensuring $\mathrm{FCR} \leq \alpha$ while reporting sets from a pre-specified informative family $\mathcal{I}$; the theoretical backbone rests on a general FCR control theorem under concordant selection and on adjusted $p$-values. The authors provide concrete instantiations for regression (excluding intervals or limiting interval length) and classification (excluding a null class or non-trivial sets), with comprehensive simulations and real-data experiments (yeast gene expression and CIFAR-10) illustrating improved power after informative selection while preserving error guarantees. The work unifies a broad class of informative constraints with conformal calibration, offering practical tools for reporting only meaningful prediction sets in high-throughput or heterogeneous-data contexts and suggesting avenues for future extensions like adaptive scoring and directional error control.

Abstract

In supervised learning, including regression and classification, conformal methods provide prediction sets for the outcome/label with finite sample coverage for any machine learning predictor. We consider here the case where such prediction sets come after a selection process. The selection process requires that the selected prediction sets be `informative' in a well defined sense. We consider both the classification and regression settings where the analyst may consider as informative only the sample with prediction sets small enough, excluding null values, or obeying other appropriate `monotone' constraints. We develop a unified framework for building such informative conformal prediction sets while controlling the false coverage rate (FCR) on the selected sample. While conformal prediction sets after selection have been the focus of much recent literature in the field, the new introduced procedures, called InfoSP and InfoSCOP, are to our knowledge the first ones providing FCR control for informative prediction sets. We show the usefulness of our resulting procedures on real and simulated data.

Figures (15)

• Figure 1: Informative prediction sets in classification for CIFAR-10 dataset, restricted to the $K=3$ classes "bird", "cat", and "dog" classes (iid setting). Informative prediction subsets are those of size at most $K-1=2$ (i.e., non-trivial, Example \ref{['ex:classif']} item 1). Selection by InfoSP are framed in red (right panel). $\alpha=10\%$. See § \ref{['sec:appli']} for more details.
• Figure 2: Informative prediction intervals when excluding $[a,b]$ (homoscedastic Gaussian regression model with perfect variance prediction), see text. The predictor $\mu$ (dotted line) does not approximate well the true $\mu^*(x)=\mathbb E[Y|X=x]$ (solid line) in the selection area (top row) and out of the selection area (bottom row). The marginal and informative prediction intervals (InfoSP and InfoSCOP) are depicted in light blue and red, respectively. While the plot corresponds to one data generation, the FCR and adjusted power computed in the title of each panel are computed with $100$ Monte-Carlo simulations. $n=1000$, $m=500$, $\alpha=0.1$.
• Figure 3: Informative prediction intervals when length-restricted (heteroscedastic Gaussian regression model with perfect mean prediction), see text. The predictor $\sigma$ under-estimates (top row) and over-estimates (bottom row) the true $\sigma^*(x)=\mathbb{V}^{1/2}[Y|X=x]$ in the selection area. The marginal and informative prediction intervals (InfoSP and InfoSCOP) are depicted in light-blue and red, respectively. While the plot corresponds to one data generation, the FCR and adjusted power computed in the title of each panel are computed with $100$ Monte-Carlo simulations. $n=1000$, $m=500$, $\alpha=0.1$.
• Figure 4: Selecting prediction sets excluding a null class in a classification setting. FCR (top row), and resolution-adjusted power (bottom row) versus SNR. The iid setting in columns 1 and 2, with balanced classes and unbalanced classes, respectively. The class-conditional setting in column 3, with a large label shift: the class probabilities are equal in the calibration sample and 0.2,0.2, and 0.6 (the null class) in the test sample. The number of data generations was 2000, 1000 data points were used for training, and $n =m = 500$. See details of the data generation in § \ref{['subsec-simul-bivariatenormal']}.
• Figure 5: FCR, average size of the selected, SR, and resolution-adjusted power for the methods, for $\alpha=0.1$.

Theorems & Definitions (48)

• Example 1.1: Informative prediction sets in regression, $\mathcal{Y}=\mathbb{R}$
• Example 1.2: Informative prediction sets in classification, $\mathcal{Y}=[K]$
• Remark 2.1
• Example 2.1: Example \ref{['ex:regression']} continued
• Example 2.2: Example \ref{['ex:classif']} continued
• Example 2.3: Combining informative subset collections
• Remark 2.2
• Remark 2.3
• Remark 2.4
• Lemma 2.1