Better Uncertainty Calibration via Proper Scores for Classification and Beyond

TL;DR

The framework of proper calibration errors is introduced, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties and can be used to reliably quantify the model calibration improvement.

Abstract

With model trustworthiness being crucial for sensitive real-world applications, practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks. Calibration errors are designed to quantify the reliability of probabilistic predictions but their estimators are usually biased and inconsistent. In this work, we introduce the framework of proper calibration errors, which relates every calibration error to a proper score and provides a respective upper bound with optimal estimation properties. This relationship can be used to reliably quantify the model calibration improvement. We theoretically and empirically demonstrate the shortcomings of commonly used estimators compared to our approach. Due to the wide applicability of proper scores, this gives a natural extension of recalibration beyond classification.

Figures (9)

• Figure 1: Estimated calibration improvement for various settings. The calibration error is estimated before and after a recalibration method (TS / ETS / DIAG) is applied and the difference (i.e. calibration improvement) is shown for increasing test set size. All common calibration estimators are sensitive with respect to the test set size and can substantially over- or underestimate the effect of performing recalibration.Only RBS robustly estimates the improvement in calibration error for all test set sizes.
• Figure 2: Estimated ECE of simulated models with perfect calibration (blue), mediocre calibration (orange), and bad calibration (green). Better calibration exacerbates ECE bias with respect to the data size, leading to unreliably calibration improvement quantification.
• Figure 3: Left: Different calibration error estimates versus the test set size of ResNet Wide 32 and CIFAR100. The red line corresponds to the square root of the Brier score which is an upper bound of the $\text{CE}_2$. The other errors are lower bounds. Right: Relative change versus data size with respect to error at full size. Averaging across a multitude of models shows a systematic trend. An unbiased estimator would give a flat line.
• Figure 4: Left: Average predicted variance throughout model training before and after recalibration. Initially, due to a bad fit, recalibration adjusts the variance accordingly for better communicated uncertainty. Once the model fit improves, the predicted variance requires less adjustment due to less uncertainty in each prediction. Middle: DSS communicates reasonably changes in the variance due to recalibration. Right: SKCE fails to capture the variance trend and behaves erratically.
• Figure 5: Different calibration error estimates versus the test set size. The red line corresponds to the square root of the Brier score which is an upper bound of $\text{CE}_2$. The other estimators are lower bounds.

Theorems & Definitions (33)

• Definition 2.1
• Theorem 3.1
• Proposition 3.2
• Proposition 3.3
• Lemma 4.1
• Definition 4.2
• Theorem 4.3
• Example 4.4
• Proposition 4.5
• Definition C.1