Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometric Convergence Analysis of Variational Inference via Bregman Divergences

Sushil Bohara, Amedeo Roberto Esposito

TL;DR

This work addresses convergence of variational inference under non-convex ELBO by developing a geometric framework based on the exponential-family Bregman divergence induced by the log-partition function $A(\phi)$. It proves that $L(\phi)=D_A(\phi^* \| \phi)$ with $\nabla L(\phi)=H(\phi)(\phi-\phi^*)$, enabling a monotonicity property and a ray-wise analysis that yields two-sided quadratic bounds via ray-wise spectral envelopes $\alpha(\phi)$ and $\beta(\phi)$. The authors derive non-asymptotic convergence rates for natural gradient descent (NGD) and Euclidean gradient descent (GD): NGD contracts along a fixed ray with rate $|1-\eta|^k$ (independent of conditioning), while GD's rate depends on the local condition number $\beta(\phi)/\alpha(\phi)$ and can be slower. Numerical experiments on Bernoulli and Gaussian VI validate the theory, showing NGD's robustness to conditioning and the advantage of the geometric approach for understanding and guiding VI optimization.

Abstract

Variational Inference (VI) provides a scalable framework for Bayesian inference by optimizing the Evidence Lower Bound (ELBO), but convergence analysis remains challenging due to the objective's non-convexity and non-smoothness in Euclidean space. We establish a novel theoretical framework for analyzing VI convergence by exploiting the exponential family structure of distributions. We express negative ELBO as a Bregman divergence with respect to the log-partition function, enabling a geometric analysis of the optimization landscape. We show that this Bregman representation admits a weak monotonicity property that, while weaker than convexity, provides sufficient structure for rigorous convergence analysis. By deriving bounds on the objective function along rays in parameter space, we establish properties governed by the spectral characteristics of the Fisher information matrix. Under this geometric framework, we prove non-asymptotic convergence rates for gradient descent algorithms with both constant and diminishing step sizes.

Geometric Convergence Analysis of Variational Inference via Bregman Divergences

TL;DR

This work addresses convergence of variational inference under non-convex ELBO by developing a geometric framework based on the exponential-family Bregman divergence induced by the log-partition function . It proves that with , enabling a monotonicity property and a ray-wise analysis that yields two-sided quadratic bounds via ray-wise spectral envelopes and . The authors derive non-asymptotic convergence rates for natural gradient descent (NGD) and Euclidean gradient descent (GD): NGD contracts along a fixed ray with rate (independent of conditioning), while GD's rate depends on the local condition number and can be slower. Numerical experiments on Bernoulli and Gaussian VI validate the theory, showing NGD's robustness to conditioning and the advantage of the geometric approach for understanding and guiding VI optimization.

Abstract

Variational Inference (VI) provides a scalable framework for Bayesian inference by optimizing the Evidence Lower Bound (ELBO), but convergence analysis remains challenging due to the objective's non-convexity and non-smoothness in Euclidean space. We establish a novel theoretical framework for analyzing VI convergence by exploiting the exponential family structure of distributions. We express negative ELBO as a Bregman divergence with respect to the log-partition function, enabling a geometric analysis of the optimization landscape. We show that this Bregman representation admits a weak monotonicity property that, while weaker than convexity, provides sufficient structure for rigorous convergence analysis. By deriving bounds on the objective function along rays in parameter space, we establish properties governed by the spectral characteristics of the Fisher information matrix. Under this geometric framework, we prove non-asymptotic convergence rates for gradient descent algorithms with both constant and diminishing step sizes.
Paper Structure (38 sections, 13 theorems, 82 equations, 4 figures)

This paper contains 38 sections, 13 theorems, 82 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Empirical Success vs. Theoretical Understanding
  5. Non-convex Optimization
  6. Preliminaries
  7. Geometric Structure of the Negative ELBO
  8. Bregman Divergence Representation
  9. Monotonicity around a point
  10. Example (Bernoulli exponential family).
  11. Ray-wise Analysis and Two-sided Bounds
  12. Ray-wise Spectral Envelopes
  13. Integral Representation
  14. Two-sided Quadratic Bounds
  15. Convergence of Natural and Euclidean Gradient Methods
  16. ...and 23 more sections

Key Result

Theorem 1

Under Assumptions 1--2, suppose the joint distribution $p(x,z)$ is exponential family in $z$ with natural parameter $\phi^*$ i.e., $p(x,z) = h(z)\exp\!(\langle\phi^*,T(z)\rangle-A(\phi^*)).$ Let $L(\phi) \;=\; \mathbb{E}_{q_\phi}\!\left[\log q_\phi(z)-\log p(x,z)\right]$ be the negative ELBO, where where $D_A$ is the Bregman divergence generated by $A$. Moreover, where $H(\phi) = \nabla^2 A(\phi

Figures (4)

  • Figure 1: Geometry of the negative ELBO for the Bernoulli example. The blue curve is $L(\phi) = D_A(\phi^{\*}\!\parallel \phi)$ with the monotonicity bound from $\phi_0$, which provides a global inequality.
  • Figure 2: Ray-wise quadratic bounds for Bernoulli exponential family. The negative ELBO $L(\phi)$ (blue) lies within adaptive bounds $\frac{\alpha(\phi)}{2}\|\phi - \phi^*\|^2 \leq L(\phi) \leq \frac{\beta(\phi)}{2}\|\phi - \phi^*\|^2$ (orange envelope)
  • Figure 3: Trajectory comparison of Natural Gradient Descent (NGD) Vs. Gradient Descent
  • Figure 4: Trajectory comparison of Natural Gradient Descent (NGD) Vs. Gradient Descent

Theorems & Definitions (35)

  • Definition 1: Exponential Family
  • Definition 2: Bregman Divergence
  • Definition 3: Gradient Descent (GD)
  • Definition 4: Natural Gradient Descent (NGD)
  • Theorem 1: Negative ELBO as a Bregman divergence
  • proof
  • Lemma 1: Bregman Three-Point Identity
  • proof
  • Theorem 2: Monotonicity around Reference point
  • proof
  • ...and 25 more