Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-dimensional adaptation of diffusion models: Convergence in total variation

Jiadong Liang, Zhihan Huang, Yuxin Chen

TL;DR

This work tackles the challenge of sampling from high-dimensional data with unknown low-dimensional structure using diffusion models. By developing total-variation convergence guarantees for the DDIM and DDPM samplers, it shows that the iteration complexity scales with the intrinsic dimension k rather than the ambient dimension d, up to logarithmic factors, under mild assumptions and with perfect score estimates. A matching lower bound clarifies the necessity of the coefficient designs originally proposed for these samplers and ties the dynamics to reverse-time ODE/SDE formulations via Tweedie’s formula. The results demonstrate genuine low-dimensional adaptivity of DDIM-type samplers and improve upon prior DDPM theories, offering practical implications for efficient, structure-aware diffusion sampling without requiring smoothness or log-concavity of the data distribution.

Abstract

This paper investigates how diffusion generative models leverage (unknown) low-dimensional structure to accelerate sampling. Focusing on two mainstream samplers -- the denoising diffusion implicit model (DDIM) and the denoising diffusion probabilistic model (DDPM) -- and assuming accurate score estimates, we prove that their iteration complexities are no greater than the order of $k/\varepsilon$ (up to some log factor), where $\varepsilon$ is the precision in total variation distance and $k$ is some intrinsic dimension of the target distribution. Our results are applicable to a broad family of target distributions without requiring smoothness or log-concavity assumptions. Further, we develop a lower bound that suggests the (near) necessity of the coefficients introduced by Ho et al.(2020) and Song et al.(2020) in facilitating low-dimensional adaptation. Our findings provide the first rigorous evidence for the adaptivity of the DDIM-type samplers to unknown low-dimensional structure, and improve over the state-of-the-art DDPM theory regarding total variation convergence.

Low-dimensional adaptation of diffusion models: Convergence in total variation

TL;DR

This work tackles the challenge of sampling from high-dimensional data with unknown low-dimensional structure using diffusion models. By developing total-variation convergence guarantees for the DDIM and DDPM samplers, it shows that the iteration complexity scales with the intrinsic dimension k rather than the ambient dimension d, up to logarithmic factors, under mild assumptions and with perfect score estimates. A matching lower bound clarifies the necessity of the coefficient designs originally proposed for these samplers and ties the dynamics to reverse-time ODE/SDE formulations via Tweedie’s formula. The results demonstrate genuine low-dimensional adaptivity of DDIM-type samplers and improve upon prior DDPM theories, offering practical implications for efficient, structure-aware diffusion sampling without requiring smoothness or log-concavity of the data distribution.

Abstract

This paper investigates how diffusion generative models leverage (unknown) low-dimensional structure to accelerate sampling. Focusing on two mainstream samplers -- the denoising diffusion implicit model (DDIM) and the denoising diffusion probabilistic model (DDPM) -- and assuming accurate score estimates, we prove that their iteration complexities are no greater than the order of (up to some log factor), where is the precision in total variation distance and is some intrinsic dimension of the target distribution. Our results are applicable to a broad family of target distributions without requiring smoothness or log-concavity assumptions. Further, we develop a lower bound that suggests the (near) necessity of the coefficients introduced by Ho et al.(2020) and Song et al.(2020) in facilitating low-dimensional adaptation. Our findings provide the first rigorous evidence for the adaptivity of the DDIM-type samplers to unknown low-dimensional structure, and improve over the state-of-the-art DDPM theory regarding total variation convergence.
Paper Structure (73 sections, 25 theorems, 213 equations, 2 tables)

This paper contains 73 sections, 25 theorems, 213 equations, 2 tables.

Table of Contents

  1. Introduction
  2. Score-based generative modeling: DDPM and DDIM
  3. Forward process.
  4. Reverse process and diffusion-based samplers.
  5. Harnessing low-dimensional structure?
  6. This paper
  7. An overview of our contributions.
  8. Notation.
  9. Preliminaries
  10. Forward process and noise schedule.
  11. Score-based generative models.
  12. ODE and SDE perspectives.
  13. Main results
  14. Key assumptions
  15. Convergence theory for DDIM
  16. ...and 58 more sections

Key Result

Theorem 1

Under Assumptions ass:low_dim-ass:ddim_score_matching, the DDIM sampler eq:DDIM-update with the coefficients $\eta_t^{\mathsf{ddim}} = \frac{1 - \alpha_t}{1 + \sqrt{\frac{\alpha_t - \overline{\alpha}_t}{1 - \overline{\alpha}_t}}}$ yields

Theorems & Definitions (26)

  • Definition 1: Covering number
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • Lemma 2
  • Corollary 1
  • Lemma 3
  • Lemma 4
  • ...and 16 more