On the Usefulness of the Fit-on-the-Test View on Evaluating Calibration of Classifiers

TL;DR

The paper introduces the fit-on-test paradigm for calibration evaluation, arguing that evaluating calibration error can be achieved by fitting a calibration map on the test data and measuring its distance to the identity. It develops two flexible calibration map families, PL and PL3, and shows how ECE can be interpreted as a fit-on-test estimator, enabling cross-validated bin selection and improved reliability diagrams. Through pseudo-real CIFAR-5m experiments, PL3 frequently best approximates the true calibration map, while PL excels when predictions are already close to calibrated, and CV-based approaches improve stability. The work also discusses limitations of fitting on the test set and suggests debiasing and broader benchmarking to further strengthen calibration evaluation in practice.

Abstract

Every uncalibrated classifier has a corresponding true calibration map that calibrates its confidence. Deviations of this idealistic map from the identity map reveal miscalibration. Such calibration errors can be reduced with many post-hoc calibration methods which fit some family of calibration maps on a validation dataset. In contrast, evaluation of calibration with the expected calibration error (ECE) on the test set does not explicitly involve fitting. However, as we demonstrate, ECE can still be viewed as if fitting a family of functions on the test data. This motivates the fit-on-the-test view on evaluation: first, approximate a calibration map on the test data, and second, quantify its distance from the identity. Exploiting this view allows us to unlock missed opportunities: (1) use the plethora of post-hoc calibration methods for evaluating calibration; (2) tune the number of bins in ECE with cross-validation. Furthermore, we introduce: (3) benchmarking on pseudo-real data where the true calibration map can be estimated very precisely; and (4) novel calibration and evaluation methods using new calibration map families PL and PL3.

Figures (15)

• Figure 1: (a) True calibration map (orange line) vs the predicted probabilities (dashed line). Connecting lines show instance-wise miscalibration. (b) Reliability diagram consists of bars (blue) with the height of average label. The red lines show the error between the mean labels and predicted probabilities in each bin. The diagrams are made with synthetic data (3000 data points, stratified sample of 50 data points from the bins shown for instance-wise errors, see Appendix \ref{['subsec:syn_experiments']} for more details).
• Figure 2: Fit-on-test estimation of calibration error: (1) a calibration map $\hat{c}$ is obtained fitting a family of calibration maps $\mathcal{C}$ by minimising the loss $l$ on the test data; (2) instance-wise calibration errors are estimated as distances of predictions from calibrated predictions; (3) overall calibration error is estimated as the average of instance-wise errors. The plots are illustrative, not based on real data.
• Figure 3: A synthetic example about how ECE can be viewed as a fit-on-test estimator of calibration error. (a) A reliability diagram of a test set with 6 instances (4 positives with predicted probabilities 0.3, 0.75, 0.85, 0.95 and 2 negatives with predicted probabilities 0.1, 0.65), yielding $ECE=0.133$; (b) a piecewise constant fit-on-test estimator yields $\widehat{CE}=0.167$; (c) a piecewise slope-1 fit-on-test estimator provably yields the same $ECE=0.133$ as the original in (a) while visualizing per-instance calibration errors also; (d) our proposed reliability diagram with diagonal filling combines elements from (a) and (c).
• Figure 4: Reliability diagrams of 3 classifiers (columns) created with different visualisation methods (rows); the concrete classification task is irrelevant. All classifiers have different ECE, but the plain reliability diagrams (top row) are identical. Differences are gradually revealed by adding bin centres indicating the average predicted probabilites (red dots, in the second row) and frequency histograms (in the third row). The last row shows our proposed reliability diagrams with diagonal filling, where ECE is better visualised because it is equal to the instance-wise average distance from the blue boundary to the main diagonal of perfect calibration.
• Figure 5: Different reliability diagrams with the number of bins or pieces optimised using cross-validation, on the same data as in Figure \ref{['fig:1_calmap_reldiag']}: (a) a reliability diagram with diagonal filling using 14 bins; (b) piecewise linear reliability diagram with 3 pieces; (c) piecewise linear in logit-logit space reliability diagram with 2 pieces.

Theorems & Definitions (8)

• Theorem 1
• Theorem 2
• proof
• Theorem 3
• Definition 1
• proof
• proof
• proof