Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models

Xuefeng Gao, Hoang M. Nguyen, Lingjiong Zhu

TL;DR

<3-5 sentence high-level summary>This work develops a rigorous convergence theory for a broad class of score-based generative models (SGMs) implemented via stochastic differential equations (SDEs). By assuming a strongly log-concave data distribution and accurate score estimation, the authors derive a 2-Wasserstein ($\mathcal{W}_{2}$) contraction bound for the reverse SDE, and decompose the error into initialization, discretization, and score-matching components to obtain iteration-complexity guarantees. The theory is specialized to Variance-Exploding (VE) and Variance-Preserving (VP) forward processes, including novel polynomial and exponential noise schedules, and is complemented by lower bounds showing limits of the approach in Gaussian and general settings. Numerical experiments on CIFAR-10 confirm that forward-process choices with lower predicted iteration complexity yield better sample quality, aligning with the theoretical predictions and highlighting the practical impact of forward-process design in SGMs.

Abstract

Score-based generative models (SGMs) is a recent class of deep generative models with state-of-the-art performance in many applications. In this paper, we establish convergence guarantees for a general class of SGMs in 2-Wasserstein distance, assuming accurate score estimates and smooth log-concave data distribution. We specialize our result to several concrete SGMs with specific choices of forward processes modelled by stochastic differential equations, and obtain an upper bound on the iteration complexity for each model, which demonstrates the impacts of different choices of the forward processes. We also provide a lower bound when the data distribution is Gaussian. Numerically, we experiment SGMs with different forward processes, some of which are newly proposed in this paper, for unconditional image generation on CIFAR-10. We find that the experimental results are in good agreement with our theoretical predictions on the iteration complexity, and the models with our newly proposed forward processes can outperform existing models.

Wasserstein Convergence Guarantees for a General Class of Score-Based Generative Models

TL;DR

<3-5 sentence high-level summary>This work develops a rigorous convergence theory for a broad class of score-based generative models (SGMs) implemented via stochastic differential equations (SDEs). By assuming a strongly log-concave data distribution and accurate score estimation, the authors derive a 2-Wasserstein () contraction bound for the reverse SDE, and decompose the error into initialization, discretization, and score-matching components to obtain iteration-complexity guarantees. The theory is specialized to Variance-Exploding (VE) and Variance-Preserving (VP) forward processes, including novel polynomial and exponential noise schedules, and is complemented by lower bounds showing limits of the approach in Gaussian and general settings. Numerical experiments on CIFAR-10 confirm that forward-process choices with lower predicted iteration complexity yield better sample quality, aligning with the theoretical predictions and highlighting the practical impact of forward-process design in SGMs.

Abstract

Score-based generative models (SGMs) is a recent class of deep generative models with state-of-the-art performance in many applications. In this paper, we establish convergence guarantees for a general class of SGMs in 2-Wasserstein distance, assuming accurate score estimates and smooth log-concave data distribution. We specialize our result to several concrete SGMs with specific choices of forward processes modelled by stochastic differential equations, and obtain an upper bound on the iteration complexity for each model, which demonstrates the impacts of different choices of the forward processes. We also provide a lower bound when the data distribution is Gaussian. Numerically, we experiment SGMs with different forward processes, some of which are newly proposed in this paper, for unconditional image generation on CIFAR-10. We find that the experimental results are in good agreement with our theoretical predictions on the iteration complexity, and the models with our newly proposed forward processes can outperform existing models.
Paper Structure (35 sections, 17 theorems, 157 equations, 4 figures, 4 tables)

This paper contains 35 sections, 17 theorems, 157 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries on SDE-Based Diffusion Models
  4. Main Results
  5. Assumptions
  6. Main Result
  7. Examples
  8. Discussions
  9. Numerical Experiments
  10. SDEs for the Forward Process
  11. Experiment Setup
  12. Empirical Results
  13. Analysis: Proofs of the Main Results
  14. Proof of Theorem \ref{['thm:discrete:2']}
  15. Completing the Proof of Theorem \ref{['thm:discrete:2']}
  16. ...and 20 more sections

Key Result

Theorem 2

Suppose that Assumptions assump:p0, assump:M:1, assump:M and assump:stepsize hold. Then, we have where and where $c(t)$ is given in c:t:defn, $\gamma_{j,\eta}$ is defined in gamma:k:defn, $c_{2}(T)$ is defined in c:2:defn and $h_{k,\eta}$ is given in h:k:eta:main.

Figures (4)

  • Figure 1: The schedules of $(\alpha_i)$ for DDPM models and $(\sigma_i)$ for NCSN (a.k.a. SMLD) models with different forward SDEs.
  • Figure 2: The FID score progressions of different SDE models on CIFAR-10
  • Figure 3: The inception score (IS) progressions of different SDE models on CIFAR-10
  • Figure 4: The FID and IS score progressions of the deep version of the best performing SDE models on CIFAR-10

Theorems & Definitions (22)

  • Remark 1
  • Theorem 2
  • Remark 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Proposition 7
  • Proposition 8
  • Lemma 9
  • Lemma 10
  • ...and 12 more