$L_2$-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Error

TL;DR

This work establishes a principled link between standard $L_2$-regularized empirical risk minimization and small smooth calibration error ($smCE$) in binary classification. It derives finite-sample bounds showing that calibration can be controlled through optimization error, regularization strength, and function-class complexity, and instantiates the theory in RKHS settings with kernel ridge regression and kernel logistic regression. The results reveal a bias-variance trade-off governed by the regularization parameter and demonstrate how calibration improves as more calibration data become available, with stronger guarantees in one-dimensional recalibration and broader, albeit slower, rates for universal kernels in higher dimensions. Empirical validation on toy and real-world recalibration tasks corroborates the theory, suggesting that well-calibrated models can be achieved without post-hoc recalibration techniques.

Abstract

Calibration of predicted probabilities is critical for reliable machine learning, yet it is poorly understood how standard training procedures yield well-calibrated models. This work provides the first theoretical proof that canonical $L_{2}$-regularized empirical risk minimization directly controls the smooth calibration error (smCE) without post-hoc correction or specialized calibration-promoting regularizer. We establish finite-sample generalization bounds for smCE based on optimization error, regularization strength, and the Rademacher complexity. We then instantiate this theory for models in reproducing kernel Hilbert spaces, deriving concrete guarantees for kernel ridge and logistic regression. Our experiments confirm these specific guarantees, demonstrating that $L_{2}$-regularized ERM can provide a well-calibrated model without boosting or post-hoc recalibration. The source code to reproduce all experiments is available at https://github.com/msfuji0211/erm_calibration.

Figures (8)

• Figure 1: Experimental Results: Effect of Sample Size and Regularization on Calibration Metrics on the toy dataset.
• Figure 2: Effect of sample size and regularization on calibration metrics on the KITTI and PCam datasets ($1$-layer neural networks).
• Figure 3: All Experimental Results of Recalibration: Effect of Recalibration Sample Size on Calibration Metrics on the KITTI dataset.
• Figure 4: All Experimental Results of Recalibration: Effect of Regularization parameter $\lambda$ on Calibration Metrics on the KITTI dataset.
• Figure 5: All Experimental Results of Recalibration: Effect of Recalibration Sample Size on Calibration Metrics on the PCam dataset.

Theorems & Definitions (27)

• Definition 1: smooth CE blasiok2023unify
• Definition 2: Post-processing gap under $\ell_{\mathrm{sq}}$ blasiok2023when
• Theorem 1: Theorem 2.4 in blasiok2023when
• Definition 3: Dual post-processing gap under $\ell^{\psi}$ blasiok2023when
• Definition 4: Dual smooth CE blasiok2023when
• Corollary 1: Corollary 2.4 in blasiok2023when
• Theorem 2: Claim 5.1 blasiok2023when
• Theorem 3
• proof : Proof outline
• Theorem 4