Adjusting Regression Models for Conditional Uncertainty Calibration

TL;DR

A novel algorithm to train a regression function to improve the conditional coverage after applying the split conformal prediction procedure is proposed and an upper bound for the miscoverage gap between the conditional coverage and the nominal coverage rate is established.

Abstract

Conformal Prediction methods have finite-sample distribution-free marginal coverage guarantees. However, they generally do not offer conditional coverage guarantees, which can be important for high-stakes decisions. In this paper, we propose a novel algorithm to train a regression function to improve the conditional coverage after applying the split conformal prediction procedure. We establish an upper bound for the miscoverage gap between the conditional coverage and the nominal coverage rate and propose an end-to-end algorithm to control this upper bound. We demonstrate the efficacy of our method empirically on synthetic and real-world datasets.

Figures (6)

• Figure 1: Coverage under Synthetic Data (Setting I) with Linear Regression, $1-\alpha=90\%$. Here we show the conditional coverage for each method. Our method can achieve the specified conditional coverage while all other methods have significantly lower conditional coverage.
• Figure 2: WSLAB for UCI Datasets with residual score across $\alpha$. Our method consistently improves WSLAB among all $\alpha$.
• Figure 3: Ablation Studies for Different Choices of $\lambda$ for Synthetic Data Setup I. We report the Marginal Coverage (MC), Conditional Coverage (CC), Set Size, and MSE for $\log(\lambda) = -1,0,1,2,3$ when $1-\alpha=90\%$.
• Figure 4: Ablation Studies for Different Choices of $\lambda$ for Synthetic Data Setup II. We report the Marginal Coverage (MC), Conditional Coverage (CC), Set Size, and MSE for $\log(\lambda) = -1,0,1,2,3$ when $1-\alpha=90\%$.
• Figure 5: Coverage under Synthetic Data (Setting II) with Linear Regression, $1-\alpha=90\%$. Here we show the conditional coverage for each method. Like most methods, KS-CP can achieve perfect conditional coverage in this case.

Theorems & Definitions (9)

• Theorem 1: Marginal Coverage Guarantee vovk2005algorithmic
• Proposition 2
• proof
• Proposition 3: Conditional Coverage Rate
• Proposition 4
• Proposition 5
• proof : Proof for \ref{['prop:ccr']}
• proof : Proof for \ref{['prop:dpi']}
• proof : Proof for \ref{['prop:asy']}