Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Variational bagging: a robust approach for Bayesian uncertainty quantification

Shitao Fan, Ilsang Ohn, David Dunson, Lizhen Lin

TL;DR

This work introduces variational bagging, a bootstrap-aggregation framework for variational Bayes that dramatically improves uncertainty quantification when mean-field VB is used or when model misspecification is present. By aggregating variational posteriors over bootstrap samples, the authors prove a Bernstein–von Mises-type result and posterior contraction guarantees, while recovering off-diagonal covariance structure that standard mean-field VB misses. The approach yields robust, well-calibrated uncertainty across parametric models, finite mixtures, deep networks, and VAEs, with demonstrated gains in predictive intervals and coverage in simulations and real data (MNIST/Omniglot). The method is computationally efficient relative to MCMC, scales to complex models, and provides practical guidance for bootstrap design, making it a versatile tool for robust Bayesian inference in modern applications.

Abstract

Variational Bayes methods are popular due to their computational efficiency and adaptability to diverse applications. In specifying the variational family, mean-field classes are commonly used, which enables efficient algorithms such as coordinate ascent variational inference (CAVI) but fails to capture parameter dependence and typically underestimates uncertainty. In this work, we introduce a variational bagging approach that integrates a bagging procedure with variational Bayes, resulting in a bagged variational posterior for improved inference. We establish strong theoretical guarantees, including posterior contraction rates for general models and a Bernstein-von Mises (BVM) type theorem that ensures valid uncertainty quantification. Notably, our results show that even when using a mean-field variational family, our approach can recover off-diagonal elements of the limiting covariance structure and provide proper uncertainty quantification. In addition, variational bagging is robust to model misspecification, with covariance structures matching those of the target covariance. We illustrate our variational bagging method in numerical studies through applications to parametric models, finite mixture models, deep neural networks, and variational autoencoders (VAEs).

Variational bagging: a robust approach for Bayesian uncertainty quantification

TL;DR

This work introduces variational bagging, a bootstrap-aggregation framework for variational Bayes that dramatically improves uncertainty quantification when mean-field VB is used or when model misspecification is present. By aggregating variational posteriors over bootstrap samples, the authors prove a Bernstein–von Mises-type result and posterior contraction guarantees, while recovering off-diagonal covariance structure that standard mean-field VB misses. The approach yields robust, well-calibrated uncertainty across parametric models, finite mixtures, deep networks, and VAEs, with demonstrated gains in predictive intervals and coverage in simulations and real data (MNIST/Omniglot). The method is computationally efficient relative to MCMC, scales to complex models, and provides practical guidance for bootstrap design, making it a versatile tool for robust Bayesian inference in modern applications.

Abstract

Variational Bayes methods are popular due to their computational efficiency and adaptability to diverse applications. In specifying the variational family, mean-field classes are commonly used, which enables efficient algorithms such as coordinate ascent variational inference (CAVI) but fails to capture parameter dependence and typically underestimates uncertainty. In this work, we introduce a variational bagging approach that integrates a bagging procedure with variational Bayes, resulting in a bagged variational posterior for improved inference. We establish strong theoretical guarantees, including posterior contraction rates for general models and a Bernstein-von Mises (BVM) type theorem that ensures valid uncertainty quantification. Notably, our results show that even when using a mean-field variational family, our approach can recover off-diagonal elements of the limiting covariance structure and provide proper uncertainty quantification. In addition, variational bagging is robust to model misspecification, with covariance structures matching those of the target covariance. We illustrate our variational bagging method in numerical studies through applications to parametric models, finite mixture models, deep neural networks, and variational autoencoders (VAEs).

Paper Structure

This paper contains 26 sections, 5 theorems, 105 equations, 14 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Model setup and notation
  4. Bernstein--von Mises (BvM) theorem for a Bayesian model
  5. Variational Bayes and asymptotic properties
  6. Asymptotic properties of BayesBag under model misspecification
  7. Variational bagging: bagged variational posterior
  8. Robust uncertainty quantification
  9. Valid variational Bayes uncertainty quantification
  10. Toy example illustration
  11. Contraction rates of the bagged variational posterior
  12. Illustrating examples
  13. Multivariate Gaussian mean
  14. Bayesian mixture models
  15. Simulation study
  16. ...and 11 more sections

Key Result

Theorem 3.1

Let $\ell_\theta(x) = \log p_{\textsc{vb}}(x \mid \theta)$, and assume the following conditions hold: Then, for $\vartheta^\dag \sim q^{\textup{bvB}}(\theta \mid X_{1:n})$, we have where and $\tilde{V}_{\textsc{vb}}^0$ is the diagonal matrix with the same diagonal entries as $V_{\textsc{vb}}^0$.

Figures (14)

  • Figure 1: $95\%$ posterior credible regions for a two-dimensional Gaussian mean: comparison of HMC, mean-field VB, and variational bagging.
  • Figure 2: $95\%$ posterior credible regions for the Gaussian mean under HMC, MFVB, and variational bagging, for varying sample size $n$ and number of bootstrap replicates $B$.
  • Figure 3: Gaussian mixture fit to $t$-mixture data.
  • Figure 4: Gaussian mixture fit to data from a double-exponential mixture model.
  • Figure 5: Gaussian mixture fit to double-exponential mixture data with larger variance.
  • ...and 9 more figures

Theorems & Definitions (8)

  • Definition 2.1
  • Theorem 3.1: Bernstein--von Mises theorem for the bagged variational posterior
  • Remark 1
  • Remark 2
  • Corollary 3.2: BvM theorem without latent variables
  • Corollary 3.3: No overconfident credible sets
  • Corollary 3.4: BvM theorem when the model is correctly specified
  • Theorem 3.5: Contraction rate