Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fisher meets Feynman: score-based variational inference with a product of experts

Diana Cai, Robert M. Gower, David M. Blei, Lawrence K. Saul

TL;DR

The paper addresses black-box variational inference by introducing a highly expressive PoE variational family where each expert is a multivariate $t$-distribution and weights are optimized to match the target density. A Feynman parameterization converts the product into a continuous Dirichlet-mixture of $t$-distributions, enabling efficient sampling and normalizing-constant computation through a latent variable representation. The authors develop a score-based VI algorithm that iteratively refines expert weights by minimizing a regularized Fisher divergence, reducing the optimization to a sequence of convex quadratic programs with provable exponential convergence to a neighborhood of the optimum. Empirically, the PoE approach better captures skewness, heavy tails, and multimodality compared with Gaussian VI and normalizing flows across synthetic targets and real posterior benchmarks, highlighting its practical impact for non-Gaussian inference tasks.

Abstract

We introduce a highly expressive yet distinctly tractable family for black-box variational inference (BBVI). Each member of this family is a weighted product of experts (PoE), and each weighted expert in the product is proportional to a multivariate $t$-distribution. These products of experts can model distributions with skew, heavy tails, and multiple modes, but to use them for BBVI, we must be able to sample from their densities. We show how to do this by reformulating these products of experts as latent variable models with auxiliary Dirichlet random variables. These Dirichlet variables emerge from a Feynman identity, originally developed for loop integrals in quantum field theory, that expresses the product of multiple fractions (or in our case, $t$-distributions) as an integral over the simplex. We leverage this simplicial latent space to draw weighted samples from these products of experts -- samples which BBVI then uses to find the PoE that best approximates a target density. Given a collection of experts, we derive an iterative procedure to optimize the exponents that determine their geometric weighting in the PoE. At each iteration, this procedure minimizes a regularized Fisher divergence to match the scores of the variational and target densities at a batch of samples drawn from the current approximation. This minimization reduces to a convex quadratic program, and we prove under general conditions that these updates converge exponentially fast to a near-optimal weighting of experts. We conclude by evaluating this approach on a variety of synthetic and real-world target distributions.

Fisher meets Feynman: score-based variational inference with a product of experts

TL;DR

The paper addresses black-box variational inference by introducing a highly expressive PoE variational family where each expert is a multivariate -distribution and weights are optimized to match the target density. A Feynman parameterization converts the product into a continuous Dirichlet-mixture of -distributions, enabling efficient sampling and normalizing-constant computation through a latent variable representation. The authors develop a score-based VI algorithm that iteratively refines expert weights by minimizing a regularized Fisher divergence, reducing the optimization to a sequence of convex quadratic programs with provable exponential convergence to a neighborhood of the optimum. Empirically, the PoE approach better captures skewness, heavy tails, and multimodality compared with Gaussian VI and normalizing flows across synthetic targets and real posterior benchmarks, highlighting its practical impact for non-Gaussian inference tasks.

Abstract

We introduce a highly expressive yet distinctly tractable family for black-box variational inference (BBVI). Each member of this family is a weighted product of experts (PoE), and each weighted expert in the product is proportional to a multivariate -distribution. These products of experts can model distributions with skew, heavy tails, and multiple modes, but to use them for BBVI, we must be able to sample from their densities. We show how to do this by reformulating these products of experts as latent variable models with auxiliary Dirichlet random variables. These Dirichlet variables emerge from a Feynman identity, originally developed for loop integrals in quantum field theory, that expresses the product of multiple fractions (or in our case, -distributions) as an integral over the simplex. We leverage this simplicial latent space to draw weighted samples from these products of experts -- samples which BBVI then uses to find the PoE that best approximates a target density. Given a collection of experts, we derive an iterative procedure to optimize the exponents that determine their geometric weighting in the PoE. At each iteration, this procedure minimizes a regularized Fisher divergence to match the scores of the variational and target densities at a batch of samples drawn from the current approximation. This minimization reduces to a convex quadratic program, and we prove under general conditions that these updates converge exponentially fast to a near-optimal weighting of experts. We conclude by evaluating this approach on a variety of synthetic and real-world target distributions.
Paper Structure (41 sections, 2 theorems, 80 equations, 11 figures, 1 table, 1 algorithm)

This paper contains 41 sections, 2 theorems, 80 equations, 11 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. A latent variable model for products of experts
  3. Products of multivariate $t$-distributions
  4. Feynman parameterization for products of experts
  5. Joint density and latent variable model
  6. Score-based VI with products of experts
  7. Selecting the experts
  8. Weighting the experts
  9. Convergence theorem
  10. Experiments
  11. Synthetic 2D targets
  12. Sinh-arcsinh target
  13. Posterior inference problems
  14. Discussion of contributions, limitations, and future work
  15. Overview
  16. ...and 26 more sections

Key Result

Theorem 3.1

Suppose that $\mathscr{D}(q;p)$ in eq:fisher is minimized by a unique $\alpha^*\!\in\!\mathcal{C}$, and also that for all $t\geq 0$ there exists some $\delta\!\geq\!0$ such that $\mathbb{E} \|\frac{1}{2} \nabla\widehat{\mathscr{D}}_{t}(\alpha^*)\| \leq \delta$ and some $\lambda\!>\!0$ such that $\ma

Figures (11)

  • Figure 1: Product vs mixture of two $t$-distributions. The experts (gray dashed curves) all have the same scale of $1$. The weights in the mixture (rightmost) are the normalized $\alpha_k$ values.
  • Figure 2: To select experts, we identify each mode and then add more experts to refine the fit.
  • Figure 3: Synthetic 2D targets. Top: Gray contours represent the target, and blue points represent samples from the fitted PoE. Bottom: The KL and Fisher divergences of each method.
  • Figure 4: 50-dimensional sinh-arcsinh target with skew and heavy tails.
  • Figure 5: posteriordb targets that highlight a range of non-Gaussian posterior properties.
  • ...and 6 more figures

Theorems & Definitions (3)

  • Theorem 3.1
  • Theorem E.1
  • proof