Conformal Risk Control for Non-Monotonic Losses

TL;DR

This work presents risk control guarantees for generic algorithms applied to possibly non-monotonic losses with multidimensional parameters and gives applications of this technique to selective image classification, FDR and IOU control of tumor segmentations, and multigroup debiasing of recidivism predictions across overlapping race and sex groups.

Abstract

Conformal risk control is an extension of conformal prediction for controlling risk functions beyond miscoverage. The original algorithm controls the expected value of a loss that is monotonic in a one-dimensional parameter. Here, we present risk control guarantees for generic algorithms applied to possibly non-monotonic losses with multidimensional parameters. The guarantees depend on the stability of the algorithm -- unstable algorithms have looser guarantees. We give applications of this technique to selective image classification, FDR and IOU control of tumor segmentations, and multigroup debiasing of recidivism predictions across overlapping race and sex groups using empirical risk minimization.

Figures (5)

• Figure 1: Cumulative average error $\bar{E}_j$ for three scenarios alongside the shrinking interval $(\alpha + (1-\alpha)/j,\; \alpha + (2-\alpha)/j]$ from Proposition \ref{['prop:ub-EK']}, with $\alpha = 0.25$ and $n = 500$. In each case $E_i \sim \mathrm{Bernoulli}(p_i)$ independently. In the well-ranked case, $p_i = \max(0,\; 2\alpha(1 - (i-1)/m))$, $m = (n+1)/2$, a linear decay from $2\alpha$ to $0$ at the midpoint, so that $\bar{E}_j$ crosses below $\alpha$ in the second half and eventually exits the interval. In the poorly ranked case, $p_i = 0.35$ for all $i$; the constant rate exceeds $\alpha$, and $\bar{E}_j$ remains above the interval. In the poorly ranked adversarial case, $p_i = \alpha$ for all $i$; the average concentrates near $\alpha$ but stays below the interval as it shrinks toward $\alpha$ from above. The blue band is the interval; the gray dotted line marks $\alpha$. $K$ is the number of $j$ for which $\bar{E}_j$ lands inside the interval.
• Figure 2: Selective classification results on Imagenet. The left panel shows the choices of thresholds from CRC, CRC-C, and LTT superimposed with the population accuracy curve. The middle panel shows a kernel density estimate of the prediction rate for all three methods over 100 resamplings of the data. The right panel shows a similar plot for the selective accuracy.
• Figure 3: FDR control results in polyp segmentation. The top left and bottom left panels show examples of polyp images and predicted masks from the PraNet, respectively. The top middle and right panels show histograms of the FDR and prediction rate for CRC-C, CRC, and LTT, over 100 resamplings of the data, respectively. Notice that CRC and CRC-C overlap almost entirely. The bottom right panel shows selections of $\hat{\theta}$ made by the three methods on the calibration data superimposed with the true FDR calculated over the whole dataset.
• Figure 4: IOU control results in polyp segmentation. The left and middle panels show histograms of the IOU and segmentation parameter $\theta$ for ERM over 100 resamplings of the data, respectively. The right panel shows the selection of $\hat{\theta}$ superimposed with the true IOU calculated over the whole dataset.
• Figure 5: Results of multigroup debiasing on COMPAS dataset. Each row is a group, and the horizontal axis shows the amount of predictive bias. Each violin plot shows the bias over 100 resamplings of the data for each method.

Theorems & Definitions (28)

• Theorem 1
• proof
• Proposition 1
• proof
• Corollary 1
• proof
• Proposition 2
• Proposition 3
• proof
• Corollary 2