Evaluating the Quality of the Quantified Uncertainty for (Re)Calibration of Data-Driven Regression Models

TL;DR

The paper addresses the critical need to reliably evaluate regression uncertainty calibration by systematically extracting a broad set of calibration metrics and benchmarking them in a model-agnostic setting. It reveals substantial inconsistencies across metrics when assessing the same model or recalibration, highlighting the risk of metric-driven cherry-picking. The study identifies ENCE and CWC as among the most dependable metrics, while also noting limitations on small datasets and hyperparameter sensitivity. Practically, the work advocates reporting multiple calibrated metrics to provide a robust view of uncertainty quality and motivates future development of principled evaluation frameworks. The findings have implications for how uncertainty estimates are validated in safety-critical applications and beyond.

Abstract

In safety-critical applications data-driven models must not only be accurate but also provide reliable uncertainty estimates. This property, commonly referred to as calibration, is essential for risk-aware decision-making. In regression a wide variety of calibration metrics and recalibration methods have emerged. However, these metrics differ significantly in their definitions, assumptions and scales, making it difficult to interpret and compare results across studies. Moreover, most recalibration methods have been evaluated using only a small subset of metrics, leaving it unclear whether improvements generalize across different notions of calibration. In this work, we systematically extract and categorize regression calibration metrics from the literature and benchmark these metrics independently of specific modelling methods or recalibration approaches. Through controlled experiments with real-world, synthetic and artificially miscalibrated data, we demonstrate that calibration metrics frequently produce conflicting results. Our analysis reveals substantial inconsistencies: many metrics disagree in their evaluation of the same recalibration result, and some even indicate contradictory conclusions. This inconsistency is particularly concerning as it potentially allows cherry-picking of metrics to create misleading impressions of success. We identify the Expected Normalized Calibration Error (ENCE) and the Coverage Width-based Criterion (CWC) as the most dependable metrics in our tests. Our findings highlight the critical role of metric selection in calibration research.

Figures (8)

• Figure 1: Structure and taxonomy of the article. The full metric names are provided in Section \ref{['sec:methods']}\ref{['sec:methods']}.
• Figure 2: Heatmap of the Spearman's rank correlation coefficients using the real-world data. The evaluation metrics label color indicates the identified metric clusters: Purple for proper scoring rules and related metrics (Group 1), teal for threshold-based metrics (Group 2), and black for others (Group 3).
• Figure 3: Normalized metric values for the deep ensembles across 16 real-world datasets. The values are normalized per metric by their mean across datasets. Groups 1 and 2 are formed based on similarity due to high correlation. Group 3 shows less/no internal consistency. Notably, metrics from different groups disagree on which model performs best on a given dataset, underscoring the substantial inconsistency in uncertainty evaluation. The IDs to the respective datasets are provided in Appendix \ref{['sec:appendix_implementation_details']}.
• Figure 4: Heatmap of the Spearman's rank correlation coefficients using synthetic data. The evaluation metrics label color indicates the previously identified metric cluster: Purple for proper scoring rules and related metrics (Group 1), teal for threshold-based metrics (Group 2), and black for others (Group 3).
• Figure 5: Normalized metric values for the deep ensembles across 10 synthetic datasets. The values are normalized per metric by their mean across datasets. The persistence of the three groups, despite the absence of aleatoric noise in the synthetic datasets, suggests that the inconsistencies are due to intrinsic differences in metric definitions. This emphasizes the need for caution when selecting metrics to evaluate uncertainty quality. The IDs to the respective datasets are provided in Appendix \ref{['sec:appendix_implementation_details']}.