PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

TL;DR

This work introduces PCS-UQ, a uncertainty quantification framework rooted in the Predictability-Computability-Stability (PCS) philosophy to address misspecification and model-selection concerns common in conformal methods. By combining a prediction-check screening step, bootstrap-driven inter-sample variability assessment, and multiplicative calibration, PCS-UQ delivers valid $1-\alpha$ prediction intervals while often reducing interval widths by around 20% compared with leading conformal baselines, and it demonstrates robust subgroup coverage. The approach scales to multi-class classification and offers practical DL approximations (dropout- and weight-noise-based perturbations) to maintain efficiency on large models, with theoretical guarantees linking modified PCS procedures to split conformal inference under exchangeability. Across 17 regression and 6 classification datasets, PCS-UQ shows favorable coverage and narrower intervals; in deep-learning settings, the approximations maintain competitive performance while dramatically reducing computation. The work also discusses extensions to data-cleaning uncertainty, binary classification, and LLMs, highlighting PCS-UQ as a versatile, principled framework for trustworthy uncertainty quantification in modern AI systems.

Abstract

As machine learning (ML) models are increasingly deployed in high-stakes domains, trustworthy uncertainty quantification (UQ) is critical for ensuring the safety and reliability of these models. Traditional UQ methods rely on specifying a true generative model and are not robust to misspecification. On the other hand, conformal inference allows for arbitrary ML models but does not consider model selection, which leads to large interval sizes. We tackle these drawbacks by proposing a UQ method based on the predictability, computability, and stability (PCS) framework for veridical data science proposed by Yu and Kumbier. Specifically, PCS-UQ addresses model selection by using a prediction check to screen out unsuitable models. PCS-UQ then fits these screened algorithms across multiple bootstraps to assess inter-sample variability and algorithmic instability, enabling more reliable uncertainty estimates. Further, we propose a novel calibration scheme that improves local adaptivity of our prediction sets. Experiments across $17$ regression and $6$ classification datasets show that PCS-UQ achieves the desired coverage and reduces width over conformal approaches by $\approx 20\%$. Further, our local analysis shows PCS-UQ often achieves target coverage across subgroups while conformal methods fail to do so. For large deep-learning models, we propose computationally efficient approximation schemes that avoid the expensive multiple bootstrap trainings of PCS-UQ. Across three computer vision benchmarks, PCS-UQ reduces prediction set size over conformal methods by $20\%$. Theoretically, we show a modified PCS-UQ algorithm is a form of split conformal inference and achieves the desired coverage with exchangeable data.

Figures (12)

• Figure 1: Comparison of PCS against Majority Vote, and two best performing conformal methods: Split conformal (XGBoost), Studentized conformal (Random Forest) across 17 datasets. We display the distribution of $\%$ improvement of PCS in the inset plot. PCS displays a significant improvement over conformal approaches.
• Figure 2: Coverage and width for PCS, and conformal regression approaches on subgroups in the Miami Housing dataset bourassa2021miami. Panels (A) and (B) demonstrate performance on subgroups formed by square footage of the house. PCS adapts width of intervals to maintain coverage across subgroups. Other conformal methods either do not achieve subgroup coverage or have larger width.
• Figure 3: Comparison of average prediction set size of PCS against best-performing conformal methods. PCS significantly reduces width across 5 out of 6 datasets.
• Figure 4: Performance of PCS with varying number of selected models over 4 datasets. The left panel displays the average $R^2$ of selected models; the right panel displays the average interval width. As the number of selected model increases, the $R^2$ decreases while the interval width increases.
• Figure 5: Performance of PCS-UQ with varying number of bootstraps over 4 datasets. The left panel displays the average interval width; the right panel displays the coverage. Both metrics stabilize after 100 bootstraps.

Theorems & Definitions (2)

• Theorem 1
• proof