Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Convergence of Score-Based Diffusion Models, Provably

Gen Li, Yu Huang, Timofey Efimov, Yuting Wei, Yuejie Chi, Yuxin Chen

TL;DR

This work addresses the slow sampling of score-based diffusion models by proposing training-free accelerated samplers for both deterministic (DDIM-like) and stochastic (DDPM-like) schemes. The authors develop a second-order-ODE-inspired accelerated deterministic sampler with a provable $O(1/\sqrt{\varepsilon})$ convergence rate (up to log factors) and a higher-order, improved stochastic sampler achieving $O(1/\varepsilon)$ convergence, under $\ell_2$ score accuracy and without requiring log-concavity or smoothness. Theoretical results include non-asymptotic TV/KL bounds and interpretation via higher-order approximations, closely related to DPM-Solver methods. Empirical results on CelebA-HQ and LSUN datasets corroborate faster convergence and crisper samples without additional training.

Abstract

Score-based diffusion models, while achieving remarkable empirical performance, often suffer from low sampling speed, due to extensive function evaluations needed during the sampling phase. Despite a flurry of recent activities towards speeding up diffusion generative modeling in practice, theoretical underpinnings for acceleration techniques remain severely limited. In this paper, we design novel training-free algorithms to accelerate popular deterministic (i.e., DDIM) and stochastic (i.e., DDPM) samplers. Our accelerated deterministic sampler converges at a rate $O(1/{T}^2)$ with $T$ the number of steps, improving upon the $O(1/T)$ rate for the DDIM sampler; and our accelerated stochastic sampler converges at a rate $O(1/T)$, outperforming the rate $O(1/\sqrt{T})$ for the DDPM sampler. The design of our algorithms leverages insights from higher-order approximation, and shares similar intuitions as popular high-order ODE solvers like the DPM-Solver-2. Our theory accommodates $\ell_2$-accurate score estimates, and does not require log-concavity or smoothness on the target distribution.

Accelerating Convergence of Score-Based Diffusion Models, Provably

TL;DR

This work addresses the slow sampling of score-based diffusion models by proposing training-free accelerated samplers for both deterministic (DDIM-like) and stochastic (DDPM-like) schemes. The authors develop a second-order-ODE-inspired accelerated deterministic sampler with a provable convergence rate (up to log factors) and a higher-order, improved stochastic sampler achieving convergence, under score accuracy and without requiring log-concavity or smoothness. Theoretical results include non-asymptotic TV/KL bounds and interpretation via higher-order approximations, closely related to DPM-Solver methods. Empirical results on CelebA-HQ and LSUN datasets corroborate faster convergence and crisper samples without additional training.

Abstract

Score-based diffusion models, while achieving remarkable empirical performance, often suffer from low sampling speed, due to extensive function evaluations needed during the sampling phase. Despite a flurry of recent activities towards speeding up diffusion generative modeling in practice, theoretical underpinnings for acceleration techniques remain severely limited. In this paper, we design novel training-free algorithms to accelerate popular deterministic (i.e., DDIM) and stochastic (i.e., DDPM) samplers. Our accelerated deterministic sampler converges at a rate with the number of steps, improving upon the rate for the DDIM sampler; and our accelerated stochastic sampler converges at a rate , outperforming the rate for the DDPM sampler. The design of our algorithms leverages insights from higher-order approximation, and shares similar intuitions as popular high-order ODE solvers like the DPM-Solver-2. Our theory accommodates -accurate score estimates, and does not require log-concavity or smoothness on the target distribution.
Paper Structure (64 sections, 17 theorems, 187 equations, 2 figures)

This paper contains 64 sections, 17 theorems, 187 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Score-based diffusion models
  3. Non-asymptotic convergence theory and acceleration
  4. Acceleration?
  5. Our contributions
  6. Other related works
  7. Convergence of score-based generative models (SGMs).
  8. Fast sampling in diffusion models.
  9. Notation
  10. Problem settings
  11. Model and sampling process
  12. Forward process.
  13. Score functions and score estimates.
  14. Data generation process.
  15. Assumptions
  16. ...and 49 more sections

Key Result

Theorem 1

Suppose that Assumptions assump:assumption-data-bounded, assumption:score-estimate and assumption:score-estimate-Jacobi hold. Then the proposed sampler eqn:ode-extragradient with the learning rate schedule eqn:alpha-t satisfies for some universal constants $C_1>0$, where we recall that $p_1$ (resp. $q_1$) denotes the distribution of $Y_1$ (resp. $X_1$).

Figures (2)

  • Figure 1: The progress of the generated samples over different numbers of NFEs (from 4 to 50), using pre-trained scores from the LSUN-Churches dataset. Top row: the vanilla DDIM-type sampler. Bottom row: the accelerated DDIM-type sampler (ours).
  • Figure 2: Examples of sampled images from the DDIM-type samplers with 5 NFEs, using pre-trained scores from the LSUN-Churches, LSUN-Bedroom, and CelebA-HQ datasets. For each dataset, the top image is the original DDIM-type sampler, and the bottom image is the accelerated DDIM-type sampler (ours).

Theorems & Definitions (19)

  • Definition 1: Score function
  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • proof
  • Lemma 5
  • Lemma 6
  • ...and 9 more