Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Theoretical guarantees for sampling and inference in generative models with latent diffusions

Belinda Tzen, Maxim Raginsky

TL;DR

The paper develops a unified stochastic-control framework for latent-diffusion generative models, showing how exact sampling and variational inference can be cast as controlled-diffusion problems and linked through a Föllmer drift construction. It provides a quantitative expressiveness guarantee: if the target terminal density is neural-net approximable, one can realize an ε-close KL approximation with a neural-net drift of controlled size, leveraging Girsanov’s theorem. An unbiased simulation scheme for expectations under latent-diffusion dynamics is also proposed, with variance bounds tied to the renewal-time distribution and its mgf, highlighting practical trade-offs for deep latent architectures. Collectively, the results illuminate the expressiveness, sampling, and unbiased estimation capabilities of diffusion-based generative models and connect them to stochastic-control and information-theoretic principles.

Abstract

We introduce and study a class of probabilistic generative models, where the latent object is a finite-dimensional diffusion process on a finite time interval and the observed variable is drawn conditionally on the terminal point of the diffusion. We make the following contributions: We provide a unified viewpoint on both sampling and variational inference in such generative models through the lens of stochastic control. We quantify the expressiveness of diffusion-based generative models. Specifically, we show that one can efficiently sample from a wide class of terminal target distributions by choosing the drift of the latent diffusion from the class of multilayer feedforward neural nets, with the accuracy of sampling measured by the Kullback-Leibler divergence to the target distribution. Finally, we present and analyze a scheme for unbiased simulation of generative models with latent diffusions and provide bounds on the variance of the resulting estimators. This scheme can be implemented as a deep generative model with a random number of layers.

Theoretical guarantees for sampling and inference in generative models with latent diffusions

TL;DR

The paper develops a unified stochastic-control framework for latent-diffusion generative models, showing how exact sampling and variational inference can be cast as controlled-diffusion problems and linked through a Föllmer drift construction. It provides a quantitative expressiveness guarantee: if the target terminal density is neural-net approximable, one can realize an ε-close KL approximation with a neural-net drift of controlled size, leveraging Girsanov’s theorem. An unbiased simulation scheme for expectations under latent-diffusion dynamics is also proposed, with variance bounds tied to the renewal-time distribution and its mgf, highlighting practical trade-offs for deep latent architectures. Collectively, the results illuminate the expressiveness, sampling, and unbiased estimation capabilities of diffusion-based generative models and connect them to stochastic-control and information-theoretic principles.

Abstract

We introduce and study a class of probabilistic generative models, where the latent object is a finite-dimensional diffusion process on a finite time interval and the observed variable is drawn conditionally on the terminal point of the diffusion. We make the following contributions: We provide a unified viewpoint on both sampling and variational inference in such generative models through the lens of stochastic control. We quantify the expressiveness of diffusion-based generative models. Specifically, we show that one can efficiently sample from a wide class of terminal target distributions by choosing the drift of the latent diffusion from the class of multilayer feedforward neural nets, with the accuracy of sampling measured by the Kullback-Leibler divergence to the target distribution. Finally, we present and analyze a scheme for unbiased simulation of generative models with latent diffusions and provide bounds on the variance of the resulting estimators. This scheme can be implemented as a deep generative model with a random number of layers.

Paper Structure

This paper contains 19 sections, 15 theorems, 140 equations.

Table of Contents

  1. Introduction and informal summary of results
  2. Method of analysis: an overview
  3. Notation
  4. Exact sampling and variational inference: a unified stochastic control viewpoint
  5. A stochastic control problem
  6. Exact sampling: the Föllmer drift
  7. Variational inference
  8. Expressiveness
  9. The proof of Theorem \ref{['thm:expressive']}
  10. Unbiased simulation
  11. The proof of Theorem \ref{['thm:log_g_cost']}
  12. Regularity properties of $f$ and the Föllmer drift
  13. Uniform approximation of the heat semigroup by a finite sum
  14. The proof of Theorem \ref{['thm:neural_net_apx']}: uniform approximation of the Föllmer drift by a neural net
  15. Proof of Theorem \ref{['thm:unbiased']}
  16. ...and 4 more sections

Key Result

Theorem 2.1

Consider the control problem eq:log_g_cost. The value function $v$ is given by where the conditional expectation is with respect to the uncontrolled diffusion process eq:generic_diffusion. Moreover, the optimal control $u^*$ is given by $u^*(x,t) = -\nabla v(x,t)$, where the gradient is taken with respect to the space variable $x \in \mathbb{R}^d$, and the corresponding contro where $p_{s,t}(\cdo

Theorems & Definitions (24)

  • Theorem 2.1
  • Remark 3.1
  • Remark 3.2
  • Theorem 3.1
  • Theorem 3.2
  • Theorem 4.1
  • Lemma 4.1
  • Remark A.1
  • Lemma B.1: Regularity of the Föllmer drift
  • proof
  • ...and 14 more