Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Score-Based Density Formula, with Applications in Diffusion Generative Models

Gen Li, Yuling Yan

TL;DR

This work provides a theoretical density formula for a continuous-time diffusion bridging a data distribution and Gaussian noise, linking the target density to the forward-process score functions. By discretizing this diffusion, the authors connect the target log-density to a sum of score-based terms used in DDPM training, and show that the ELBO minimizer nearly recovers the true KL objective as the number of steps grows. The results furnish a solid theoretical foundation for using ELBO-based training in diffusion models, explain the role of score-matching regularization in GANs, justify ELBO use in diffusion classifiers, and illuminate the diffusion loss in autoregressive contexts. Overall, the density formula provides a unifying lens for understanding why ELBO-based objectives work across diffusion-based generative methods and related learning paradigms.

Abstract

Score-based generative models (SGMs) have revolutionized the field of generative modeling, achieving unprecedented success in generating realistic and diverse content. Despite empirical advances, the theoretical basis for why optimizing the evidence lower bound (ELBO) on the log-likelihood is effective for training diffusion generative models, such as DDPMs, remains largely unexplored. In this paper, we address this question by establishing a density formula for a continuous-time diffusion process, which can be viewed as the continuous-time limit of the forward process in an SGM. This formula reveals the connection between the target density and the score function associated with each step of the forward process. Building on this, we demonstrate that the minimizer of the optimization objective for training DDPMs nearly coincides with that of the true objective, providing a theoretical foundation for optimizing DDPMs using the ELBO. Furthermore, we offer new insights into the role of score-matching regularization in training GANs, the use of ELBO in diffusion classifiers, and the recently proposed diffusion loss.

A Score-Based Density Formula, with Applications in Diffusion Generative Models

TL;DR

This work provides a theoretical density formula for a continuous-time diffusion bridging a data distribution and Gaussian noise, linking the target density to the forward-process score functions. By discretizing this diffusion, the authors connect the target log-density to a sum of score-based terms used in DDPM training, and show that the ELBO minimizer nearly recovers the true KL objective as the number of steps grows. The results furnish a solid theoretical foundation for using ELBO-based training in diffusion models, explain the role of score-matching regularization in GANs, justify ELBO use in diffusion classifiers, and illuminate the diffusion loss in autoregressive contexts. Overall, the density formula provides a unifying lens for understanding why ELBO-based objectives work across diffusion-based generative methods and related learning paradigms.

Abstract

Score-based generative models (SGMs) have revolutionized the field of generative modeling, achieving unprecedented success in generating realistic and diverse content. Despite empirical advances, the theoretical basis for why optimizing the evidence lower bound (ELBO) on the log-likelihood is effective for training diffusion generative models, such as DDPMs, remains largely unexplored. In this paper, we address this question by establishing a density formula for a continuous-time diffusion process, which can be viewed as the continuous-time limit of the forward process in an SGM. This formula reveals the connection between the target density and the score function associated with each step of the forward process. Building on this, we demonstrate that the minimizer of the optimization objective for training DDPMs nearly coincides with that of the true objective, providing a theoretical foundation for optimizing DDPMs using the ELBO. Furthermore, we offer new insights into the role of score-matching regularization in training GANs, the use of ELBO in diffusion classifiers, and the recently proposed diffusion loss.
Paper Structure (30 sections, 3 theorems, 134 equations)

This paper contains 30 sections, 3 theorems, 134 equations.

Table of Contents

  1. Introduction
  2. Problem set-up
  3. Denoising diffusion probabilistic model
  4. A continuous-time SDE for the forward process
  5. The score-based density formula
  6. Main results
  7. From continuous time to discrete time
  8. Comparison with other results
  9. Implications
  10. Certifying the validity of optimizing ELBO in DDPM
  11. Understanding the role of regularization in GAN
  12. Confirming the use of ELBO in diffusion classifier
  13. Demystifying the diffusion loss in autoregressive models
  14. Proof of Theorem \ref{['thm:main']}
  15. Proof of Claim \ref{['claim-1']}.
  16. ...and 15 more sections

Key Result

Theorem 1

Consider the diffusion process $(X_{t})_{0\leq t<1}$ defined in (eq:diffusion-SDE), and let $\rho_{t}$ be the density of $X_{t}$. For any $0<t_{1}<t_{2}<1$, we have

Theorems & Definitions (6)

  • Theorem 1
  • Proposition 1
  • Proposition 2
  • Claim 1
  • Claim 2
  • Claim 3