Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

Minshuo Chen, Kaixuan Huang, Tuo Zhao, Mengdi Wang

TL;DR

The paper provides a principled theory for diffusion models when data lie on an unknown low-dimensional linear subspace. It introduces an encoder-decoder score network that achieves universal L2 approximation of the score, and proves sample-efficient score estimation with rates depending on the intrinsic dimension rather than ambient dimension. By analyzing the backward diffusion in the latent subspace and leveraging Girsanov’s theorem, the authors establish distribution-estimation guarantees, including subspace recovery and controlled convergence to the latent data distribution while the orthogonal component vanishes. The results demonstrate that diffusion models can circumvent the ambient-dimensionality curse and effectively capture intrinsic geometric structure through end-to-end learning. The framework lays groundwork for extending diffusion theory to broader manifold settings and motivates end-to-end subspace-aware generative modeling.

Abstract

Diffusion models achieve state-of-the-art performance in various generation tasks. However, their theoretical foundations fall far behind. This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. Our result provides sample complexity bounds for distribution estimation using diffusion models. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. Furthermore, the generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution. The convergence rate depends on the subspace dimension, indicating that diffusion models can circumvent the curse of data ambient dimensionality.

Score Approximation, Estimation and Distribution Recovery of Diffusion Models on Low-Dimensional Data

TL;DR

The paper provides a principled theory for diffusion models when data lie on an unknown low-dimensional linear subspace. It introduces an encoder-decoder score network that achieves universal L2 approximation of the score, and proves sample-efficient score estimation with rates depending on the intrinsic dimension rather than ambient dimension. By analyzing the backward diffusion in the latent subspace and leveraging Girsanov’s theorem, the authors establish distribution-estimation guarantees, including subspace recovery and controlled convergence to the latent data distribution while the orthogonal component vanishes. The results demonstrate that diffusion models can circumvent the ambient-dimensionality curse and effectively capture intrinsic geometric structure through end-to-end learning. The framework lays groundwork for extending diffusion theory to broader manifold settings and motivates end-to-end subspace-aware generative modeling.

Abstract

Diffusion models achieve state-of-the-art performance in various generation tasks. However, their theoretical foundations fall far behind. This paper studies score approximation, estimation, and distribution recovery of diffusion models, when data are supported on an unknown low-dimensional linear subspace. Our result provides sample complexity bounds for distribution estimation using diffusion models. We show that with a properly chosen neural network architecture, the score function can be both accurately approximated and efficiently estimated. Furthermore, the generated distribution based on the estimated score function captures the data geometric structures and converges to a close vicinity of the data distribution. The convergence rate depends on the subspace dimension, indicating that diffusion models can circumvent the curse of data ambient dimensionality.
Paper Structure (51 sections, 20 theorems, 240 equations, 4 figures)

This paper contains 51 sections, 20 theorems, 240 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related work
  3. Preliminaries
  4. Forward and backward SDEs
  5. Score matching
  6. Score decomposition
  7. Score approximation and estimation
  8. Score approximation
  9. Universal approximation under the $L^2$ norm
  10. Lipschitz score network
  11. Time as an additional input dimension
  12. Score estimation theory
  13. Distribution estimation
  14. Subspace recovery error
  15. Tradeoff on $t_0$
  16. ...and 36 more sections

Key Result

Lemma 1

Let data $\mathbf{x} = A\mathbf{z}$ follows Assumption assumption:subspace_data. The score function $\nabla \log p_t(\mathbf{x})$ decomposes as where with $\phi_t( \cdot | \mathbf{z})$ being the Gaussian density function of ${\sf N}(\alpha(t)\mathbf{z}, h(t)I_d)$ for $\alpha(t) = e^{-t/2}$ and $h(t) = 1 - e^{-t}$.

Figures (4)

  • Figure 1: Demonstration of score decomposition induces two backward processes.
  • Figure 2: Network architecture of ${\mathcal{S}}_{\rm NN}$.
  • Figure 3: Construction of $\bar{\mathbf{f}}_{\bm{\theta}}(\mathbf{z}, t)$ for approximating $\mathbf{g}(\mathbf{z}, t)$. For a fixed $t$, inside $[-R, R]^d$, we uniformly partition the hypercube into smaller hypercubes. On each of the small hypercube, we locally approximate $\mathbf{g}(\mathbf{z}, t)$ by its value on the center. To detect the local region, we construct a trapezoid function $\psi$ on each coordinate.
  • Figure 4: Trapezoid function in one dimension.

Theorems & Definitions (22)

  • Lemma 1
  • Example 1
  • Remark 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • ...and 12 more