Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

Anand Jerry George, Nicolas Macris

Abstract

We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.

Asymptotic Learning Curves for Diffusion Models with Random Features Score and Manifold Data

Abstract

We study the theoretical behavior of denoising score matching--the learning task associated to diffusion models--when the data distribution is supported on a low-dimensional manifold and the score is parameterized using a random feature neural network. We derive asymptotically exact expressions for the test, train, and score errors in the high-dimensional limit. Our analysis reveals that, for linear manifolds the sample complexity required to learn the score function scales linearly with the intrinsic dimension of the manifold, rather than with the ambient dimension. Perhaps surprisingly, the benefits of low-dimensional structure starts to diminish once we have a non-linear manifold. These results indicate that diffusion models can benefit from structured data; however, the dependence on the specific type of structure is subtle and intricate.
Paper Structure (26 sections, 9 theorems, 89 equations, 4 figures)

This paper contains 26 sections, 9 theorems, 89 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Generative modelling
  3. Context of This Work
  4. Our Contributions
  5. Related Works
  6. Preliminaries
  7. Diffusion Models
  8. Score model: Random Features Neural Network
  9. Data Model: Hidden Manifold Model
  10. Main Results
  11. Evaluation Metrics: Test, Train, and Score Errors
  12. Test and Train Errors for the Optimal RFNN Score
  13. Test Error for Exact Score
  14. Score error
  15. Discussion
  16. ...and 11 more sections

Key Result

Lemma 1

Let $f:\mathbb{R}\xspace\to\mathbb{R}\xspace$ be any smooth function such that $\mathbb{E}_{g\sim\mathcal{N}\left(0,1\right)}{\mleft[f(g)\mright]} = 0$. Let $\phi_i = \frac{w_i^T}{\sqrt{d}}\sigma\left(\frac{M}{\sqrt{D}} \xi\right)$ and $\phi_i' = \frac{w_i^T}{\sqrt{d}}\left(\nu_1\frac{M}{\sqrt{D}} \

Figures (4)

  • Figure 1: Test (solid lines) and train (dashed lines) errors for $\varrho =$ ReLU, $\sigma(x)=x$. Dotted horizontal lines denote the test error for exact score function obtained using \ref{['eqn:exact_score_formula']}.
  • Figure 2: Score errors for linear and non-linear manifolds. Solid lines are for $t=0.001$, and dashed lines are for $t=0.1$.
  • Figure 3: Score error and sample complexity for $\varrho =$ ReLU, $\sigma(x)=x$.
  • Figure 4: Comparison of test (solid lines) and train (dashed lines) errors with numerical simulations (points) for $\varrho =$ ReLU, $\sigma=$ tanh.

Theorems & Definitions (13)

  • Definition 1
  • Lemma 1
  • Theorem 2
  • Lemma 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • proof
  • Lemma 6
  • Lemma
  • ...and 3 more