Optimized Certainty Equivalent Risk-Controlling Prediction Sets

TL;DR

Optimized certainty equivalent RCPS (OCE-RCPS) is introduced, a novel framework that provides high-probability guarantees on general optimized certainty equivalent (OCE) risk measures, including conditional value-at-risk (CVaR) and entropic risk.

Abstract

In safety-critical applications such as medical image segmentation, prediction systems must provide reliability guarantees that extend beyond conventional expected loss control. While risk-controlling prediction sets (RCPS) offer probabilistic guarantees on the expected risk, they fail to capture tail behavior and worst-case scenarios that are crucial in high-stakes settings. This paper introduces optimized certainty equivalent RCPS (OCE-RCPS), a novel framework that provides high-probability guarantees on general optimized certainty equivalent (OCE) risk measures, including conditional value-at-risk (CVaR) and entropic risk. OCE-RCPS leverages upper confidence bounds to identify prediction set parameters that satisfy user-specified risk tolerance levels with provable reliability. We establish theoretical guarantees showing that OCE-RCPS satisfies the desired probabilistic constraint for loss functions such as miscoverage and false negative rate. Experiments on image segmentation demonstrate that OCE-RCPS consistently meets target satisfaction rates across various risk measures and reliability configurations, while OCE-CRC fails to provide probabilistic guarantees.

Figures (3)

• Figure 1: Given an input $x$, e.g., an image, and a pre-trained model $p(y|x)$, optimized certainty equivalent conformal risk control (OCE-CRC) yeh2025conformal and the proposed OCE risk-controlling prediction sets (OCE-RCPS) find a hyperparameter configuration $\hat{\lambda}$, e.g., an inclusion threshold, such that the reliability constraint $R_{\text{OCE}}(\hat{\lambda}) \leq \alpha$ is satisfied for any given OCE risk measure $R_{\text{OCE}}(\cdot)$, e.g., the conditional value-at-risk (CVaR). OCE-CRC yeh2025conformal ensures this condition only on average with respect to the calibration data used to optimize the hyperparameter $\lambda$. In contrast, OCE-RCPS ensures that the condition $R_{\text{OCE}}(\hat{\lambda}) \leq \alpha$ holds with probability no smaller than a target level $1-\delta$. The left panel of the figure shows representative results for the CVaR of the false negative rate (FNR) loss in a segmentation task pogorelov2017kvasir, with white, red, and green pixels denoting correct predictions, false negatives, and false positives, respectively. The right panel of the figure compares the empirical distribution of the CVaR and of the relative prediction set size with targets $\alpha=0.2$ and $1-\delta=0.8$. While OCE-CRC yields hyperparameters $\hat{\lambda}$ violating the requirement $R_{\text{OCE}}(\hat{\lambda}) \leq \alpha$ with probability larger than the requirement $\delta=0.2$, the proposed OCE-RCPS meets the target satisfaction rate $1-\delta=0.8$.
• Figure 2: Distribution of entropic risk (top) and relative prediction set size (bottom) for OCE-CRC and OCE-RCPS, with tolerated risk level $\alpha=0.2$, target satisfaction rate $1-\delta = 0.8$, and entropic risk sensitivity level $\beta=3$. Dashed lines in the bottom panel indicate the median relative prediction set size for each method.
• Figure 3: Average satisfaction rate (top) and relative prediction set size (bottom) for OCE-CRC and OCE-RCPS across varying target satisfaction rates $1-\delta \in \{ 0.6, 0.7, 0.8,0.9 \}$, with tolerated risk level $\alpha=0.4$ and CVaR concentration level $\beta=0.9$. Red dashed lines in the top panel indicate the corresponding target levels $1-\delta$.