Bayesian posterior approximation with stochastic ensembles

TL;DR

This work addresses the computational challenge of Bayesian posterior inference for neural networks by introducing stochastic ensembles that merge deep ensembles with stochastic regularization within a Bayesian variational framework. The authors formulate a general variational family that includes Monte Carlo dropout, DropConnect, and a novel non-parametric dropout (SE3), and they show these ensembles can better approximate the true posterior $p(\theta|\mathcal{D})$ than standard baselines. Through experiments on a toy problem and CIFAR-10/100 with a ResNet-20-FRN, they compare against Hamiltonian Monte Carlo posteriors and demonstrate that stochastic ensembles—especially SE1 (Monte Carlo dropout)—achieve closer alignment to HMC in metrics like predictive log-likelihood, accuracy, calibration, and robustness to distribution shifts, while SE3 performs best on simpler tasks. The results suggest substantial gains in uncertainty quantification and reliability for deep Bayesian inference, with practical training advantages over some SGMCMC methods.

Abstract

We introduce ensembles of stochastic neural networks to approximate the Bayesian posterior, combining stochastic methods such as dropout with deep ensembles. The stochastic ensembles are formulated as families of distributions and trained to approximate the Bayesian posterior with variational inference. We implement stochastic ensembles based on Monte Carlo dropout, DropConnect and a novel non-parametric version of dropout and evaluate them on a toy problem and CIFAR image classification. For both tasks, we test the quality of the posteriors directly against Hamiltonian Monte Carlo simulations. Our results show that stochastic ensembles provide more accurate posterior estimates than other popular baselines for Bayesian inference.

Figures (4)

• Figure 1: (a-c) Three different datasets of training data and corresponding to them HMC uncertainty estimates, entropy and mutual information.
• Figure 2: Histogram plots of predictive uncertainty computed using different ensemble methods. The HMC baselines are shown in orange in every subplot for better visualisation contrast.
• Figure 3: Calibration curves associated with different ensembles trained and evaluated on CIFAR.
• Figure 4: Evaluation on CIFAR-C. The CIFAR test sets are corrupted in 16 different ways at various intensities on the scale of 1 to 5. The error bars depict min and max of the corresponding metrics over each corruption, with the boxes indicating quartiles.