Model uncertainty quantification using feature confidence sets for outcome excursions

TL;DR

This work introduces a model-agnostic framework for uncertainty quantification in prediction tasks by constructing inner and outer feature confidence sets that certify whether a subset of test features yields outcomes above a threshold. It shifts uncertainty evaluation from prediction-level bands to feature-space excursions, providing formal containment guarantees for the true excursion set with finite-sample and asymptotic assurances. The method relies on bootstrap-based predictions to form data-dependent confidence sets and introduces inflation-based finite-sample bounds when boundary points are sparse, with rigorous theorems and corollaries. Empirical validation on simulations (both correctly specified and misspecified models) and real data applications in housing pricing and sepsis timing demonstrates improved precision and sensitivity over baseline interval-inversion methods, highlighting practical utility for risk-aware decision making in high-stakes settings.

Abstract

When implementing prediction models for high-stakes real-world applications such as medicine, finance, and autonomous systems, quantifying prediction uncertainty is critical for effective risk management. Traditional approaches to uncertainty quantification, such as confidence and prediction intervals, provide probability coverage guarantees for the expected outcomes $f(\boldsymbol{x})$ or the realized outcomes $f(\boldsymbol{x})+ε$. Instead, this paper introduces a novel, model-agnostic framework for quantifying uncertainty in continuous and binary outcomes using confidence sets for outcome excursions, where the goal is to identify a subset of the feature space where the expected or realized outcome exceeds a specific value. The proposed method constructs data-dependent inner and outer confidence sets that aim to contain the true feature subset for which the expected or realized outcomes of these features exceed a specified threshold. We establish theoretical guarantees for the probability that these confidence sets contain the true feature subset, both asymptotically and for finite sample sizes. The framework is validated through simulations and applied to real-world datasets, demonstrating its utility in contexts such as housing price prediction and time to sepsis diagnosis in healthcare. This approach provides a unified method for uncertainty quantification that is broadly applicable across various continuous and binary prediction models.

Figures (6)

• Figure 1: Confidence set construction workflow. A. Given a training data set $(X,Y)$ and a pre-defined model structure, $B$ non-parametric bootstrap training datasets are used to train $B$ models with different weights. Each of the trained models predicts the outcomes for the unseen $m$ test feature points $\mathcal{X}_m$, obtaining $B$ set of predictions. The user specifies the excursion level $c$ and target coverage probability lower bound $1-\alpha$. Based on the $B$ sets of predictions on the $m$ test feature points, the confidence set construction algorithm outputs the estimated inner set $\mathrm{CS}^{i}_{c}$ and the outer set $\mathrm{CS}^{o}_{c}$ with probability guarantees (above the target lower-bound and below the upper-bound estimated by the algorithm) that the true targeted subset is sandwiched within the two estimated sets. The estimated outer set $\mathrm{CS}^{o}_{c}$ (green, yellow and red region) contains the unknown target subset $\mathcal{X}_m(c)$ (red and yellow region). Subsequently, the target subset contains the estimated inner set $\mathrm{CS}^{i}_{c}$ (red region). We are confident that the red points belong to $\mathcal{X}_m(c)$, while the green points remain uncertain. If the assumptions hold, our theorem and corollary guarantee this containment statement is lower and upper bounded.
• Figure 2: Demonstration of confidence sets constructed for 500 testing points in a 1D space, using a two-layer neural network model trained on 100 training samples where the true function is a cosine function. Upper plot: the expected outcome $f(x)$ (solid black line), predictions for 500 points (gray), and the realized outcomes $y$ (red). Left plot: confidence sets for the expected outcome $f(x)$ at the level $c=2$. The inner set $\mathrm{CS}^{i}_{c}$ (red) and the outer set $\mathrm{CS}^{o}_{c}$ (red + green) are labeled on the point predictions; the complement of the outer set is labeled in blue. Right plot: confidence sets for the realized outcome $y$ with similar labeling.
• Figure 3: Validation of \ref{['algorithm_confidence_set']} under correct model specification with linear data: confidence set (CS, blue), two naive algorithms by inverting point-wise confidence interval (CI, red) and inverting simultaneous confidence interval (SCI, green)—across different numbers of covariates $p$ (3, 6, and 10) and various training sample sizes $n$ (100, 200, 400, 800). The x-axis represents the training sample size, while the y-axis on the left indicates the coverage rate probability. The dashed horizontal lines represent the Targeted Lower Bound (TLB), with specific values of TLB = 0.6 and TLB = 0.9. The error bars displays the Estimated Upper Bound (EUB), which should be above the observed coverage rate. The blue line denotes the number of points in the inflated boundary for confidence set algorithm, corresponding to the right y-axis. The expected and realized outcomes are shown across the rows.
• Figure 4: Validation of \ref{['algorithm_confidence_set']} under correct model specification with logistic data: confidence set (CS, blue), two reference algorithms by inverting point-wise confidence interval (CI, red) and inverting simultaneous confidence interval (SCI, green)—across different numbers of covariates $p$ (3, 6, and 10) and various training sample sizes $n$ (100, 200, 400, 800). The x-axis represents the training sample size, while the y-axis on the left indicates the coverage rate probability. The dashed horizontal lines represent the Targeted Lower Bound (TLB), with specific values of TLB = 0.6 and TLB = 0.9. The error bars displays the Estimated Upper Bound (EUB), which should be above the observed coverage rate. The blue line denote the number of points in the inflated boundary for the confidence set algorithm, corresponding to the right y-axis.
• Figure 5: Validation of \ref{['algorithm_confidence_set']} under misspecified model with non-linear data: confidence set (CS, blue), two naive algorithms by inverting point-wise confidence interval (CI, red) and inverting simultaneous confidence interval (SCI, green)—across different training sample sizes $n$ (100, 200, 400, 800). The x-axis represents the training sample size, while the y-axis on the left indicates the coverage rate probability. The dashed horizontal lines represent the Targeted Lower Bound (TLB), with specific values of TLB = 0.6 and TLB = 0.9. The error bars displays the Estimated Upper Bound (EUB), which should be above the observed coverage rate. The blue line denote the number of points in the inflated boundary for the confidence set algorithm, corresponding to the right y-axis. The result of neural network model and XGBoost model are shown across the columns. The expected and realized outcome are shown across the rows.

Theorems & Definitions (9)

• Theorem 1
• Theorem 2
• Corollary 1
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof