Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Adaptive Exponential Integration for Stable Gaussian Mixture Black-Box Variational Inference

Baojun Che, Yifan Chen, Daniel Zhengyu Huang, Xinying Mao, Weijie Wang

TL;DR

The paper tackles instability in black-box variational inference with Gaussian mixtures by developing an affine-invariant natural-gradient flow combined with an adaptive exponential integrator that preserves positive definiteness of covariance matrices. An adaptive time-stepping scheme with a warm-up and a convergence phase, plus annealing-based initialization and a geometry-driven interpretation via mirror descent on the SPD manifold, yield stable and efficient updates without requiring derivatives of the target. Theoretical results establish exponential convergence in the noise-free Gaussian setting and almost-sure convergence under Monte Carlo noise, while empirical tests on multimodal targets, Neal's funnel, and a Darcy flow inverse problem demonstrate robust performance, fast convergence, and effective multimodal recovery. The approach offers a gradient-free, geometry-aware BBVI framework with practical scheduling and initialization strategies that scale to moderate-dimensional problems. The work connects Gaussian-mixture BBVI to manifold optimization and shows how adaptive integration and affine invariance improve stability, making Gaussian-mixture BBVI a practical tool for complex Bayesian inference in scientific computing.

Abstract

Black-box variational inference (BBVI) with Gaussian mixture families offers a flexible approach for approximating complex posterior distributions without requiring gradients of the target density. However, standard numerical optimization methods often suffer from instability and inefficiency. We develop a stable and efficient framework that combines three key components: (1) affine-invariant preconditioning via natural gradient formulations, (2) an exponential integrator that unconditionally preserves the positive definiteness of covariance matrices, and (3) adaptive time stepping to ensure stability and to accommodate distinct warm-up and convergence phases. The proposed approach has natural connections to manifold optimization and mirror descent. For Gaussian posteriors, we prove exponential convergence in the noise-free setting and almost-sure convergence under Monte Carlo estimation, rigorously justifying the necessity of adaptive time stepping. Numerical experiments on multimodal distributions, Neal's multiscale funnel, and a PDE-based Bayesian inverse problem for Darcy flow demonstrate the effectiveness of the proposed method.

Adaptive Exponential Integration for Stable Gaussian Mixture Black-Box Variational Inference

TL;DR

The paper tackles instability in black-box variational inference with Gaussian mixtures by developing an affine-invariant natural-gradient flow combined with an adaptive exponential integrator that preserves positive definiteness of covariance matrices. An adaptive time-stepping scheme with a warm-up and a convergence phase, plus annealing-based initialization and a geometry-driven interpretation via mirror descent on the SPD manifold, yield stable and efficient updates without requiring derivatives of the target. Theoretical results establish exponential convergence in the noise-free Gaussian setting and almost-sure convergence under Monte Carlo noise, while empirical tests on multimodal targets, Neal's funnel, and a Darcy flow inverse problem demonstrate robust performance, fast convergence, and effective multimodal recovery. The approach offers a gradient-free, geometry-aware BBVI framework with practical scheduling and initialization strategies that scale to moderate-dimensional problems. The work connects Gaussian-mixture BBVI to manifold optimization and shows how adaptive integration and affine invariance improve stability, making Gaussian-mixture BBVI a practical tool for complex Bayesian inference in scientific computing.

Abstract

Black-box variational inference (BBVI) with Gaussian mixture families offers a flexible approach for approximating complex posterior distributions without requiring gradients of the target density. However, standard numerical optimization methods often suffer from instability and inefficiency. We develop a stable and efficient framework that combines three key components: (1) affine-invariant preconditioning via natural gradient formulations, (2) an exponential integrator that unconditionally preserves the positive definiteness of covariance matrices, and (3) adaptive time stepping to ensure stability and to accommodate distinct warm-up and convergence phases. The proposed approach has natural connections to manifold optimization and mirror descent. For Gaussian posteriors, we prove exponential convergence in the noise-free setting and almost-sure convergence under Monte Carlo estimation, rigorously justifying the necessity of adaptive time stepping. Numerical experiments on multimodal distributions, Neal's multiscale funnel, and a PDE-based Bayesian inverse problem for Darcy flow demonstrate the effectiveness of the proposed method.
Paper Structure (22 sections, 4 theorems, 89 equations, 7 figures)

This paper contains 22 sections, 4 theorems, 89 equations, 7 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Preliminaries and literature review
  4. Natural gradient
  5. Black-box variational inference
  6. Organization
  7. Stable Gaussian mixture BBVI
  8. Natural gradient variational inference
  9. Adaptive exponential integration
  10. Theoretical insights
  11. Convergence study
  12. Relation to manifold optimization and mirror descent
  13. Implementation Details
  14. Scheduler
  15. Annealing-based initialization
  16. ...and 7 more sections

Key Result

Theorem 3.1

Assume that $\Delta t_{\rm max} \leq 1$, $\beta \leq 1$, and that the initial covariance matrix $\Sigma_0$ is symmetric positive definite with minimum and maximum eigenvalues $\lambda_{\rm min}(\Sigma_0)$ and $\lambda_{\rm max}(\Sigma_0)$. Given an error tolerance $\epsilon < 1$, for iterations eq:G where the hidden constant in the $\mathcal{O}(\cdot)$ depends only on $\Delta t_{\max}$ and $\beta$

Figures (7)

  • Figure 1: Multi-dimensional model problems: Cases A to C are arranged from the top row to the bottom row. Each panel shows the reference marginal density along with the GMBBVI estimates for problem dimensions 2, 10, 50 (from left to right). The projected means of each Gaussian component are marked by red circles, and the projected means at iteration $0$ (after annealing) are marked by grey crosses. The fifth panel displays the total variation distance between the reference marginal density and the estimated marginal densities across the iterations, with shaded area representing the standard deviation computed from 10 independent trials.
  • Figure 1: Sensitivity study for different choices of initial condition (first row), number of modes $K$ (second row), scheduler $\eta$ (third row) and annealing strategy (last row) in the 10-dimensional Case A. The fifth panel reports the TV distance between the reference marginal density and the estimated marginal densities across iterations, where solid line represents the mean, and shaded area represents the standard deviation computed from 10 independent trials.
  • Figure 2: Neal's funnel model problem: Dimensions $2$, $10$, and $50$ are arranged from the top row to the bottom row. Each panel shows the reference marginal density of $(\theta_{(1)},\theta_{(2)})$ along with the GMBBVI and WALNUTS estimates (from left to right). For GMBBVI, the projected means of each Gaussian component are marked by red circles, and the initializations are marked by grey crosses. For WALNUTS, red dots denote the last 5000 particles, and the marginal density is visualized using kernel density estimation. The fourth panel displays the reference marginal density of $\theta_{(1)}$ along with the GMBBVI and WALNUTS estimates. The fifth panel reports convergence metrics for $\theta_{(1)}$, including its mean and variance, for both methods, as well as the total variation distance between the reference marginal density and the GMBBVI estimates of $(\theta_{(1)},\theta_{(2)})$, with the shaded area representing the standard deviation computed from 10 independent trials.
  • Figure 3: Darcy flow problem: The pressure field and symmetry observations at 120 equidistant points (solid black dots). Their mirroring points are marked (empty black dots).
  • Figure 4: Darcy flow problem: The true initial permeability field (leftmost), its mirrored field (middle-left), recovered permeability fields by modes close to truth (middle-right) and close to mirrored truth (rightmost).
  • ...and 2 more figures

Theorems & Definitions (11)

  • Theorem 3.1
  • Remark 3.2
  • Remark 3.3
  • Theorem 3.4
  • Theorem 3.5
  • Remark 3.6
  • Proof 1: Proof of \ref{['theorem-Sigma_n-no-noise']}
  • Proof 2: Proof of \ref{['thm:Sigma_n-noise']}
  • Lemma B.1
  • Proof 3
  • ...and 1 more