Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Investigating Batch Inference in a Sequential Monte Carlo Framework for Neural Networks

Andrew Millard, Joshua Murphy, Peter Green, Simon Maskell

TL;DR

This paper tackles Bayesian neural network posterior inference by applying data-annealed Sequential Monte Carlo (SMC) to enable mini-batch likelihood and gradient evaluations while preserving a particle-based posterior approximation. It compares gradient-based Markov kernels, notably Hamiltonian Monte Carlo $\text{(HMC)}$ and Langevin dynamics $\text{(LD)}$, and analyzes several data-annealing schemes, including a Shannon-entropy–guided approach $\text{SDA}$, across image-classification benchmarks. The results show that data-annealed SMC can achieve up to roughly $6\times$ faster training with only modest accuracy loss, with Constant-to-refine (CTR) offering a strong practical trade-off between speed and accuracy; HMC generally outperforms LD. The work suggests that cheaper, progressive data inclusion can substantially accelerate Bayesian NN posterior sampling and identifies directions for adaptive kernel length and schemes in future research $\left(\text{e.g., adaptive trajectory lengths and more sophisticated annealing} ight)$.

Abstract

Bayesian inference allows us to define a posterior distribution over the weights of a generic neural network (NN). Exact posteriors are usually intractable, in which case approximations can be employed. One such approximation - variational inference - is computationally efficient when using mini-batch stochastic gradient descent as subsets of the data are used for likelihood and gradient evaluations, though the approach relies on the selection of a variational distribution which sufficiently matches the form of the posterior. Particle-based methods such as Markov chain Monte Carlo and Sequential Monte Carlo (SMC) do not assume a parametric family for the posterior by typically require higher computational cost. These sampling methods typically use the full-batch of data for likelihood and gradient evaluations, which contributes to this computational expense. We explore several methods of gradually introducing more mini-batches of data (data annealing) into likelihood and gradient evaluations of an SMC sampler. We find that we can achieve up to $6\times$ faster training with minimal loss in accuracy on benchmark image classification problems using NNs.

Investigating Batch Inference in a Sequential Monte Carlo Framework for Neural Networks

TL;DR

This paper tackles Bayesian neural network posterior inference by applying data-annealed Sequential Monte Carlo (SMC) to enable mini-batch likelihood and gradient evaluations while preserving a particle-based posterior approximation. It compares gradient-based Markov kernels, notably Hamiltonian Monte Carlo and Langevin dynamics , and analyzes several data-annealing schemes, including a Shannon-entropy–guided approach , across image-classification benchmarks. The results show that data-annealed SMC can achieve up to roughly faster training with only modest accuracy loss, with Constant-to-refine (CTR) offering a strong practical trade-off between speed and accuracy; HMC generally outperforms LD. The work suggests that cheaper, progressive data inclusion can substantially accelerate Bayesian NN posterior sampling and identifies directions for adaptive kernel length and schemes in future research .

Abstract

Bayesian inference allows us to define a posterior distribution over the weights of a generic neural network (NN). Exact posteriors are usually intractable, in which case approximations can be employed. One such approximation - variational inference - is computationally efficient when using mini-batch stochastic gradient descent as subsets of the data are used for likelihood and gradient evaluations, though the approach relies on the selection of a variational distribution which sufficiently matches the form of the posterior. Particle-based methods such as Markov chain Monte Carlo and Sequential Monte Carlo (SMC) do not assume a parametric family for the posterior by typically require higher computational cost. These sampling methods typically use the full-batch of data for likelihood and gradient evaluations, which contributes to this computational expense. We explore several methods of gradually introducing more mini-batches of data (data annealing) into likelihood and gradient evaluations of an SMC sampler. We find that we can achieve up to faster training with minimal loss in accuracy on benchmark image classification problems using NNs.
Paper Structure (6 sections, 17 equations, 2 figures, 1 table)

This paper contains 6 sections, 17 equations, 2 figures, 1 table.

Table of Contents

  1. Introduction
  2. Sequential Monte Carlo Samplers
  3. Markov and L-Kernels
  4. Batch Inference and Data Annealing
  5. Annealing Schemes
  6. Discussion and Conclusions

Figures (2)

  • Figure 1: Visualisation of batching schemes.
  • Figure 2: Change in $\beta$ values and batch count over iterations for a single run on each dataset with the SDA scheme. Early on, $\beta_k$ often resets to 1, rapidly adding MBs, but the inclusion rate slows over time.