Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Theory and computation for structured variational inference

Shunan Sheng, Bohan Wu, Bennett Zhu, Sinho Chewi, Aram-Alexandre Pooladian

TL;DR

This work proves the first results for existence, uniqueness, and self-consistency of the variational approximation, and derives quantitative approximation error bounds for the variational approximation to the posterior, extending prior work from the mean-field setting to the star-structured setting.

Abstract

Structured variational inference constitutes a core methodology in modern statistical applications. Unlike mean-field variational inference, the approximate posterior is assumed to have interdependent structure. We consider the natural setting of star-structured variational inference, where a root variable impacts all the other ones. We prove the first results for existence, uniqueness, and self-consistency of the variational approximation. In turn, we derive quantitative approximation error bounds for the variational approximation to the posterior, extending prior work from the mean-field setting to the star-structured setting. We also develop a gradient-based algorithm with provable guarantees for computing the variational approximation using ideas from optimal transport theory. We explore the implications of our results for Gaussian measures and hierarchical Bayesian models, including generalized linear models with location family priors and spike-and-slab priors with one-dimensional debiasing. As a by-product of our analysis, we develop new stability results for star-separable transport maps which might be of independent interest.

Theory and computation for structured variational inference

TL;DR

This work proves the first results for existence, uniqueness, and self-consistency of the variational approximation, and derives quantitative approximation error bounds for the variational approximation to the posterior, extending prior work from the mean-field setting to the star-structured setting.

Abstract

Structured variational inference constitutes a core methodology in modern statistical applications. Unlike mean-field variational inference, the approximate posterior is assumed to have interdependent structure. We consider the natural setting of star-structured variational inference, where a root variable impacts all the other ones. We prove the first results for existence, uniqueness, and self-consistency of the variational approximation. In turn, we derive quantitative approximation error bounds for the variational approximation to the posterior, extending prior work from the mean-field setting to the star-structured setting. We also develop a gradient-based algorithm with provable guarantees for computing the variational approximation using ideas from optimal transport theory. We explore the implications of our results for Gaussian measures and hierarchical Bayesian models, including generalized linear models with location family priors and spike-and-slab priors with one-dimensional debiasing. As a by-product of our analysis, we develop new stability results for star-separable transport maps which might be of independent interest.

Paper Structure

This paper contains 27 sections, 43 theorems, 324 equations, 2 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related work
  4. Notation
  5. A first theory for star-structured variational inference
  6. Existence and uniqueness of the minimizer
  7. Regularity and approximation quality via self-consistency equations
  8. Examples
  9. Gaussian posterior
  10. Bayesian generalized linear models with various priors
  11. Computational guarantees via optimal transport
  12. From star-structured measures to star-separable maps
  13. Existence and uniqueness of the minimizer via (strong) convexity
  14. Theoretical properties of star-separable maps
  15. A projected gradient descent algorithm for solving \ref{['KR-obj']}
  16. ...and 12 more sections

Key Result

Proposition 2.1

The following equivalence holds:

Figures (2)

  • Figure 1: Bayesian GLMs with different structural dependencies on the model parameters.
  • Figure 2: A general purpose illustration of star-structured variational inference.

Theorems & Definitions (84)

  • Proposition 2.1: Dynamic programming
  • Theorem 2.2: Existence and uniqueness of the SSVI minimizer
  • Remark 2.3
  • Theorem 2.4: Self-consistency equations for $\pi^\star$
  • Remark 2.5
  • Lemma 2.6
  • Corollary 2.7: Uniqueness
  • Theorem 2.8: Log-concavity and log-smoothness of $\pi^\star$
  • Theorem 2.9: Approximation guarantee for SSVI
  • Theorem 2.10
  • ...and 74 more