Manifold Generalization Provably Proceeds Memorization in Diffusion Models

Abstract

Diffusion models often generate novel samples even when the learned score is only \emph{coarse} -- a phenomenon not accounted for by the standard view of diffusion training as density estimation. In this paper, we show that, under the \emph{manifold hypothesis}, this behavior can instead be explained by coarse scores capturing the \emph{geometry} of the data while discarding the fine-scale distributional structure of the population measure~$μ_{\scriptscriptstyle\mathrm{data}}$. Concretely, whereas estimating the full data distribution $μ_{\scriptscriptstyle\mathrm{data}}$ supported on a $k$-dimensional manifold is known to require the classical minimax rate $\tilde{\mathcal{O}}(N^{-1/k})$, we prove that diffusion models trained with coarse scores can exploit the \emph{regularity of the manifold support} and attain a near-parametric rate toward a \emph{different} target distribution. This target distribution has density uniformly comparable to that of~$μ_{\scriptscriptstyle\mathrm{data}}$ throughout any $\tilde{\mathcal{O}}\bigl(N^{-β/(4k)}\bigr)$-neighborhood of the manifold, where $β$ denotes the manifold regularity. Our guarantees therefore depend only on the smoothness of the underlying support, and are especially favorable when the data density itself is irregular, for instance non-differentiable. In particular, when the manifold is sufficiently smooth, we obtain that \emph{generalization} -- formalized as the ability to generate novel, high-fidelity samples -- occurs at a statistical rate strictly faster than that required to estimate the full population distribution~$μ_{\scriptscriptstyle\mathrm{data}}$.

Figures (5)

• Figure 1: Geometry precedes memorization in diffusion training.Top row: training dynamics across three regimes. The manifold error (dark, left axis) decreases rapidly, while the memorization rate (light, right axis) stays low for coarsely optimized scores. The "generalization" window is the regime where both manifold error and memorization are small. Bottom row: our diagnostic for manifold learning. Alongside the training loss (dark, left axis), we report the mean alignment (light, right axis) between the learned score $s^\theta$ and the projection direction, $\langle \mathop{\mathrm{\mathrm{Proj}_{\mathcal{M}}}}\nolimits, s^\theta\rangle/(\|\mathop{\mathrm{\mathrm{Proj}_{\mathcal{M}}}}\nolimits\|\,\|s^\theta\|)$. Across regimes, alignment rises quickly and saturates early, suggesting that the coarse score network first recovers manifold geometry, while memorization is a later-stage effect.
• Figure C.1: A local representation of a submanifold ${\mathcal{M}}\in \mathcal{C}^\beta$.
• Figure D.1: Understanding the hypothesis score function class $\{s_\eta\}$ in the local coordinate: i) pick a reference point $x_{\mathrm{ref}} \in {\mathcal{M}^\star}$; ii) any $k$-dimensional $\mathcal{C}^{\beta-1}$ submanifold ${\mathcal{M}_\eta}$ passing $x_{\mathrm{ref}}$ can be parameterized by $[W_\eta, \hat{N}_\eta]$ in the sense that for all $\hat{x} \in {\mathcal{M}_\eta} \cap {B^{\textup{Euc}}_{D}}(x_{\mathrm{ref}}, h)$, there exists a unique coordinate $(\hat{u}, \hat{N}_\eta(\hat{u}))$ under the basis $(W_\eta, W_\eta^\perp)$; iii) for any $x \in {B^{\textup{Euc}}_{D}}(x_{\mathrm{ref}}, h)$, the projection onto ${\mathcal{M}_\eta}$ is unique, denoted by $\pi_\eta(x)$; iv) the score function indexed by $\eta$ can be written as $s_\eta(t, x) := - \frac{x - \pi_\eta(x)}{t}$ for $x \in {B^{\textup{Euc}}_{D}}(x_{\mathrm{ref}}, h)$.
• Figure D.2: Change of basis. For any point $\hat{x} \in {\mathcal{M}_\eta} \cap {B^{\textup{Euc}}_{D}}(x_{\mathrm{ref}}, h)$, use $(\hat{u}, \hat{N}_\eta(\hat{u}))$ and $(u, N_\eta(u))$ to denote its coordinates under the bases $(W_\eta, W_\eta^\perp)$ and $(W_{\mathrm{ref}}, W_{\mathrm{ref}}^\perp)$ respectively. When $\sigma_{\min}(W_\eta^\top W_{\mathrm{ref}}) >0$, for a sufficiently small $h > 0$, one can identify $N_\eta$ with $\hat{N}_\eta$ up to a diffeomorphism.
• Figure :

Theorems & Definitions (55)

• theorem 1: Main; informal
• theorem 2: Large-noise reduction
• theorem 3: Hausdorff recovery and projection accuracy
• lemma 1: Contraction to an $\varepsilon$-tube
• lemma 2: Tangential drift bound
• definition 1: Covering
• theorem 4: Coverage of the population surrogate
• proof : proof of \ref{['thm:large_noise_reduction']}
• theorem B.1: Weaker version of \ref{['thm:hausdorff_and_projection']}
• remark 1