Structured Matrix Scaling for Multi-Class Calibration
Eugène Berta, David Holzmüller, Michael I. Jordan, Francis Bach
TL;DR
The paper tackles miscalibration in multiclass probability estimates and argues for expressive post-hoc calibration beyond temperature scaling. By deriving a theoretically motivated quadratic softmax calibration and introducing structured regularization (SMS and SVS), it adapts model complexity to the calibration data to avoid overfitting. The authors provide an efficient open-source probmetrics implementation using SAGA optimization, accompanied by a principled hyperparameter grid search and meta-learning guidance. Extensive experiments across tabular and computer vision benchmarks show that the proposed methods consistently improve calibration, particularly as the number of classes increases, offering a practical and scalable alternative to existing calibration techniques.
Abstract
Post-hoc recalibration methods are widely used to ensure that classifiers provide faithful probability estimates. We argue that parametric recalibration functions based on logistic regression can be motivated from a simple theoretical setting for both binary and multiclass classification. This insight motivates the use of more expressive calibration methods beyond standard temperature scaling. For multi-class calibration however, a key challenge lies in the increasing number of parameters introduced by more complex models, often coupled with limited calibration data, which can lead to overfitting. Through extensive experiments, we demonstrate that the resulting bias-variance tradeoff can be effectively managed by structured regularization, robust preprocessing and efficient optimization. The resulting methods lead to substantial gains over existing logistic-based calibration techniques. We provide efficient and easy-to-use open-source implementations of our methods, making them an attractive alternative to common temperature, vector, and matrix scaling implementations.