Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Humble your Overconfident Networks: Unlearning Overfitting via Sequential Monte Carlo Tempered Deep Ensembles

Andrew Millard, Zheng Zhao, Joshua Murphy, Simon Maskell

TL;DR

The paper presents a scalable post‑training Bayesian refinement for pretrained neural networks by embedding Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) proposals into Sequential Monte Carlo (SMC) samplers. This nonparametric posterior sampling approach mitigates overfitting from SGD, improves calibration, and enhances uncertainty quantification for image classification, OOD detection, and transfer learning, outperforming SGD and competitive Bayesian baselines in several metrics. The method leverages mini‑batch gradients for proposal dynamics while using full‑dataset likelihoods for weight updates, and employs tempered or cold posterior targets to avoid particle degeneracy, with a warm‑up and SGD‑initialization strategy to accelerate convergence. Practically, this yields well‑calibrated Bayesian neural networks that better capture predictive uncertainty, translating into more reliable OOD detection and transfer performance, albeit with increased computational cost and sensitivity to long sampling runs.

Abstract

Sequential Monte Carlo (SMC) methods offer a principled approach to Bayesian uncertainty quantification but are traditionally limited by the need for full-batch gradient evaluations. We introduce a scalable variant by incorporating Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) proposals into SMC, enabling efficient mini-batch based sampling. Our resulting SMCSGHMC algorithm outperforms standard stochastic gradient descent (SGD) and deep ensembles across image classification, out-of-distribution (OOD) detection, and transfer learning tasks. We further show that SMCSGHMC mitigates overfitting and improves calibration, providing a flexible, scalable pathway for converting pretrained neural networks into well-calibrated Bayesian models.

Humble your Overconfident Networks: Unlearning Overfitting via Sequential Monte Carlo Tempered Deep Ensembles

TL;DR

The paper presents a scalable post‑training Bayesian refinement for pretrained neural networks by embedding Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) proposals into Sequential Monte Carlo (SMC) samplers. This nonparametric posterior sampling approach mitigates overfitting from SGD, improves calibration, and enhances uncertainty quantification for image classification, OOD detection, and transfer learning, outperforming SGD and competitive Bayesian baselines in several metrics. The method leverages mini‑batch gradients for proposal dynamics while using full‑dataset likelihoods for weight updates, and employs tempered or cold posterior targets to avoid particle degeneracy, with a warm‑up and SGD‑initialization strategy to accelerate convergence. Practically, this yields well‑calibrated Bayesian neural networks that better capture predictive uncertainty, translating into more reliable OOD detection and transfer performance, albeit with increased computational cost and sensitivity to long sampling runs.

Abstract

Sequential Monte Carlo (SMC) methods offer a principled approach to Bayesian uncertainty quantification but are traditionally limited by the need for full-batch gradient evaluations. We introduce a scalable variant by incorporating Stochastic Gradient Hamiltonian Monte Carlo (SGHMC) proposals into SMC, enabling efficient mini-batch based sampling. Our resulting SMCSGHMC algorithm outperforms standard stochastic gradient descent (SGD) and deep ensembles across image classification, out-of-distribution (OOD) detection, and transfer learning tasks. We further show that SMCSGHMC mitigates overfitting and improves calibration, providing a flexible, scalable pathway for converting pretrained neural networks into well-calibrated Bayesian models.
Paper Structure (39 sections, 68 equations, 9 figures, 6 tables, 3 algorithms)

This paper contains 39 sections, 68 equations, 9 figures, 6 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Sequential Monte Carlo Samplers
  3. Stochastic Gradient Hamiltonian Monte Carlo
  4. Applying SMC Samplers to Bayesian Neural Networks
  5. Stochastic Gradient Hamiltonian Dynamics for Mini-Batches
  6. Entropy-Preserving SGHMC Dynamics
  7. Scalable Proposals for Bayesian Neural Networks
  8. Warm-up
  9. Targeting Tempered Posteriors
  10. Leveraging SGD Initialization and Comparison to Other Bayesian Methods
  11. Experiments
  12. Synthetic Multimodal Data
  13. Bayesian Neural Networks for Image Classification
  14. Bayesian Neural Networks for Out-of-Distribution Detection
  15. Transfer Learning with the STL Dataset
  16. ...and 24 more sections

Figures (9)

  • Figure 1: Left: 25 Mode GMM Distribution which we are targeting. Center: Samples generated by the SMC sampler with a HMC proposal, a total of 200,000 samples were collected. Right: Kernel Density Estimate of the posterior distribution based off the samples generated.
  • Figure 2: Training and validation loss for MNIST.
  • Figure 3: Training and validation loss for FashionMNIST.
  • Figure 4: Training and validation loss for CIFAR-10.
  • Figure 5: Training and validation loss for CIFAR-100.
  • ...and 4 more figures