Reassessing How to Compare and Improve the Calibration of Machine Learning Models

TL;DR

This work critically reevaluates how calibration is reported in machine learning, showing that relying solely on metrics like ECE and accuracy can be misled by trivial recalibration strategies. It advances a theoretical framework based on Bregman divergences to decompose generalization into a sharpness term and a calibration term, and introduces calibration-sharpness diagrams that jointly visualize calibration and generalization alongside the density of predicted confidences. The authors prove that confidence calibration provides a valid lower bound to full calibration within this framework and demonstrate the consistency of a kernel-regression estimator used in the visualization, applying the approach to large-scale vision models to reveal trade-offs between calibration and sharpness. The practical contribution is a principled, density-aware visualization and reporting protocol (with open-source tooling) that enables robust comparison of recalibration methods and prevents over-optimistic conclusions about calibration improvements.

Abstract

A machine learning model is calibrated if its predicted probability for an outcome matches the observed frequency for that outcome conditional on the model prediction. This property has become increasingly important as the impact of machine learning models has continued to spread to various domains. As a result, there are now a dizzying number of recent papers on measuring and improving the calibration of (specifically deep learning) models. In this work, we reassess the reporting of calibration metrics in the recent literature. We show that there exist trivial recalibration approaches that can appear seemingly state-of-the-art unless calibration and prediction metrics (i.e. test accuracy) are accompanied by additional generalization metrics such as negative log-likelihood. We then use a calibration-based decomposition of Bregman divergences to develop a new extension to reliability diagrams that jointly visualizes calibration and generalization error, and show how our visualization can be used to detect trade-offs between calibration and generalization. Along the way, we prove novel results regarding the relationship between full calibration error and confidence calibration error for Bregman divergences. We also establish the consistency of the kernel regression estimator for calibration error used in our visualization approach, which generalizes existing consistency results in the literature.

Figures (14)

• Figure 1: Proposed calibration-sharpness diagrams using MSE/Brier score for the experiments of Section \ref{['sec:pitfalls']}. Y-axes are not aligned here due to the large sharpness gap of MRR.
• Figure 2: Comparing calibration-sharpness diagrams to reliability diagrams using histogram binning and isotonic regression. Y-axes of HB and IR are aligned for more direct comparison.
• Figure 3: Calibration-sharpness diagrams using MSE/Brier score for the ResNet-50 experiments of Table \ref{['tab:mrr2']}.
• Figure 4: Calibration-sharpness diagrams using MSE/Brier score for the EfficientNet experiments of Table \ref{['tab:mrr3']}.
• Figure 5: Calibration-sharpness diagrams using MSE/Brier score for the ConvNeXt experiments of Table \ref{['tab:mrr4']}.

Theorems & Definitions (29)

• Remark 3.1
• Lemma 4.0
• Proposition 4.0
• Proposition 4.0
• Proposition 4.0
• Proposition 4.0
• Lemma 4.0
• Corollary 4.0
• Proposition 4.0
• Theorem 4.1