Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Good is the Bayes Posterior in Deep Neural Networks Really?

Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub Świątkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, Sebastian Nowozin

TL;DR

This work demonstrates through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD and argues that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

Abstract

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

How Good is the Bayes Posterior in Deep Neural Networks Really?

TL;DR

This work demonstrates through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD and argues that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

Abstract

During the past five years the Bayesian deep learning community has developed increasingly accurate and efficient approximate inference procedures that allow for Bayesian inference in deep neural networks. However, despite this algorithmic progress and the promise of improved uncertainty quantification and sample efficiency there are---as of early 2020---no publicized deployments of Bayesian neural networks in industrial practice. In this work we cast doubt on the current understanding of Bayes posteriors in popular deep neural networks: we demonstrate through careful MCMC sampling that the posterior predictive induced by the Bayes posterior yields systematically worse predictions compared to simpler methods including point estimates obtained from SGD. Furthermore, we demonstrate that predictive performance is improved significantly through the use of a "cold posterior" that overcounts evidence. Such cold posteriors sharply deviate from the Bayesian paradigm but are commonly used as heuristic in Bayesian deep learning papers. We put forward several hypotheses that could explain cold posteriors and evaluate the hypotheses through experiments. Our work questions the goal of accurate posterior approximations in Bayesian deep learning: If the true Bayes posterior is poor, what is the use of more accurate approximations? Instead, we argue that it is timely to focus on understanding the origin of the improved performance of cold posteriors.

Paper Structure

This paper contains 69 sections, 2 theorems, 51 equations, 40 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Bayesian Deep Learning
  3. Why Should Bayes ($T=1$) be Better?
  4. Cold Posteriors Perform Better
  5. Deep Learning Models: ResNet-20 and LSTM
  6. Why is a Temperature of $T < 1$ a Problem?
  7. Confirmation from the Literature
  8. Related work that uses $T < 1$ posteriors in SG-MCMC.
  9. Related work that uses $T < 1$ posteriors in Variational Bayes.
  10. Bayesian Deep Learning in Practice
  11. Posterior Simulation using Langevin Dynamics
  12. Stochastic Gradient MCMC (SG-MCMC)
  13. Accurate SG-MCMC Simulation
  14. Inference: Is it Accurate?
  15. Hypothesis: Inaccurate SDE Simulation
  16. ...and 54 more sections

Key Result

Proposition 1

Given a Hamiltonian $H({\boldsymbol{\theta}},{\mathbf{m}})$ corresponding to Langevin dynamics, and an arbitrary smooth vector field $B: \mathbb{R}^d \times \mathbb{R}^d \to \mathbb{R}^d \times \mathbb{R}^d$ satisfying then

Figures (40)

  • Figure 1: The "cold posterior" effect: for a ResNet-20 on CIFAR-10 we can improve the generalization performance significantly by cooling the posterior with a temperature $T \ll 1$, deviating from the Bayes posterior $p({\boldsymbol{\theta}}|\mathcal{D}) \propto \exp(-U({\boldsymbol{\theta}})/T)$ at $T=1$.
  • Figure 2: Predictive performance on the CIFAR-10 test set for a cooled ResNet-20 Bayes posterior. The SGD baseline is separately tuned for the same model (Appendix \ref{['sec:resnet-sgd-details']}).
  • Figure 3: Predictive performance on the IMDB sentiment task test set for a tempered CNN-LSTM Bayes posterior. Error bars are $\pm$ one standard error over three runs. See Appendix \ref{['sec:cnnlstm-sgd-details']}.
  • Figure 4: Symplectic Euler Langevin scheme.
  • Figure 5: HMC (left) agrees closely with SG-MCMC (right) for synthetic data on multilayer perceptrons. A star indicates the optimal temperature for each model: for the synthetic data sampled from the prior there are no cold posteriors and both sampling methods perform best at $T=1$.
  • ...and 35 more figures

Theorems & Definitions (3)

  • Proposition 1: Constructing Temperature Observables
  • Proposition 2: Jensen Prior
  • proof