Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Calibration-Aware Bayesian Learning

Jiayi Huang, Sangwoo Park, Osvaldo Simeone

TL;DR

The paper tackles unreliable uncertainty estimates in deep models by focusing on calibration. It introduces Calibration-Aware Bayesian Neural Networks (CA-BNNs), which jointly apply a data-dependent calibration regularizer and a data-independent prior regularizer within a variational Bayesian framework, enabling ensemble-based epistemic uncertainty while improving calibration. Central contributions include a calibration-aware VI objective, differentiable calibration measures (notably WMMCE-based AECE) with differentiable gradient schemes, and empirical validation on 20 Newsgroups and CIFAR-10 showing improved ECE and reliability diagrams. The work demonstrates that combining calibration-aware regularization with Bayesian learning yields better-calibrated decisions under practical model misspecification, with potential impacts on deploying reliable ML systems in safety-critical domains.

Abstract

Deep learning models, including modern systems like large language models, are well known to offer unreliable estimates of the uncertainty of their decisions. In order to improve the quality of the confidence levels, also known as calibration, of a model, common approaches entail the addition of either data-dependent or data-independent regularization terms to the training loss. Data-dependent regularizers have been recently introduced in the context of conventional frequentist learning to penalize deviations between confidence and accuracy. In contrast, data-independent regularizers are at the core of Bayesian learning, enforcing adherence of the variational distribution in the model parameter space to a prior density. The former approach is unable to quantify epistemic uncertainty, while the latter is severely affected by model misspecification. In light of the limitations of both methods, this paper proposes an integrated framework, referred to as calibration-aware Bayesian neural networks (CA-BNNs), that applies both regularizers while optimizing over a variational distribution as in Bayesian learning. Numerical results validate the advantages of the proposed approach in terms of expected calibration error (ECE) and reliability diagrams.

Calibration-Aware Bayesian Learning

TL;DR

The paper tackles unreliable uncertainty estimates in deep models by focusing on calibration. It introduces Calibration-Aware Bayesian Neural Networks (CA-BNNs), which jointly apply a data-dependent calibration regularizer and a data-independent prior regularizer within a variational Bayesian framework, enabling ensemble-based epistemic uncertainty while improving calibration. Central contributions include a calibration-aware VI objective, differentiable calibration measures (notably WMMCE-based AECE) with differentiable gradient schemes, and empirical validation on 20 Newsgroups and CIFAR-10 showing improved ECE and reliability diagrams. The work demonstrates that combining calibration-aware regularization with Bayesian learning yields better-calibrated decisions under practical model misspecification, with potential impacts on deploying reliable ML systems in safety-critical domains.

Abstract

Deep learning models, including modern systems like large language models, are well known to offer unreliable estimates of the uncertainty of their decisions. In order to improve the quality of the confidence levels, also known as calibration, of a model, common approaches entail the addition of either data-dependent or data-independent regularization terms to the training loss. Data-dependent regularizers have been recently introduced in the context of conventional frequentist learning to penalize deviations between confidence and accuracy. In contrast, data-independent regularizers are at the core of Bayesian learning, enforcing adherence of the variational distribution in the model parameter space to a prior density. The former approach is unable to quantify epistemic uncertainty, while the latter is severely affected by model misspecification. In light of the limitations of both methods, this paper proposes an integrated framework, referred to as calibration-aware Bayesian neural networks (CA-BNNs), that applies both regularizers while optimizing over a variational distribution as in Bayesian learning. Numerical results validate the advantages of the proposed approach in terms of expected calibration error (ECE) and reliability diagrams.
Paper Structure (15 sections, 18 equations, 4 figures, 1 algorithm)

This paper contains 15 sections, 18 equations, 4 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. BACKGROUND
  3. Frequentist and Bayesian learning
  4. Frequentist Learning
  5. Bayesian Learning
  6. Expected Calibration Error
  7. CALIBRATION-AWARE FREQUENTIST LEARNING
  8. CALIBRATION-AWARE BAYESIAN LEARNING
  9. DIFFERENTIABLE CALIBRATION MEASURES
  10. WMMCE
  11. Gradient of the WMMCE
  12. EXPERIMENTS AND DISCUSSION
  13. 20 Newsgroups
  14. CIFAR-10
  15. Conclusion

Figures (4)

  • Figure 1: ECE and accuracy as a function of number of epochs for 20 Newsgroups classification task for FNN, CA-FNN kumar2018trainable, and the modified CA-FNN introduced in Sec. \ref{['differentiable_section']} with fully differentiable batch regularizer. The shaded areas correspond to $75\%$ intervals of the realized values.
  • Figure 2: ECE and accuracy as a function of hyperparameter $\lambda$ on 20 Newsgroups data set for FNN, BNN, CA-FNN and CA-BNN both with fully differentiable batch regularizer. The shaded areas correspond to $60\%$ intervals of the realized values.
  • Figure 3: Reliability diagrams for the CIFAR-10 classification task given the predictor trained by (i) FNN (top-left), (ii) BNN (top-right), (iii) CA-FNN (bottom-left), (iv) CA-BNN (bottom-right).
  • Figure : CA-BNN Training Procedure