Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold

Ye He, Yitong Qiu, Molei Tao

TL;DR

This work tackles how diffusion models generalize when not memorizing the training data by introducing a data-dependent, ridge-based geometry described by time-indexed log-density ridge sets $\mathcal{R}_t$. It formalizes the inference as a reach–align–slide process around these ridges and proves stage-wise dynamical properties (entry into a ridge tube, normal alignment, and tangent sliding) that shape generation. By decomposing training-induced errors into normal and tangent components and applying this to a random-feature neural network (RFNN) formalism, the paper links architecture and optimization to generation biases, yielding concrete predictions on when inter-mode generation arises. Empirical validation on synthetic 2D distributions and MNIST latent spaces supports the theory, showing that normalization toward ridges dominates early inference and tangential sliding governs the spread along ridges, with weight schedules and initialization modulating these effects.

Abstract

When a diffusion model is not memorizing the training data set, how does it generalize exactly? A quantitative understanding of the distribution it generates would be beneficial to, for example, an assessment of the model's performance for downstream applications. We thus explicitly characterize what diffusion model generates, by proposing a log-density ridge manifold and quantifying how the generated data relate to this manifold as inference dynamics progresses. More precisely, inference undergoes a reach-align-slide process centered around the ridge manifold: trajectories first reach a neighborhood of the manifold, then align as being pushed toward or away from the manifold in normal directions, and finally slide along the manifold in tangent directions. Within the scope of this general behavior, different training errors will lead to different normal and tangent motions, which can be quantified, and these detailed motions characterize when inter-mode generations emerge. More detailed understanding of training dynamics will lead to more accurate quantification of the generation inductive bias, and an example of random feature model will be considered, for which we can explicitly illustrate how diffusion model's inductive biases originate as a composition of architectural bias and training accuracy, and how they evolve with the inference dynamics. Experiments on synthetic multimodal distributions and MNIST latent diffusion support the predicted directional effects, in both low- and high-dimensions.

Diffusion Model's Generalization Can Be Characterized by Inductive Biases toward a Data-Dependent Ridge Manifold

TL;DR

This work tackles how diffusion models generalize when not memorizing the training data by introducing a data-dependent, ridge-based geometry described by time-indexed log-density ridge sets . It formalizes the inference as a reach–align–slide process around these ridges and proves stage-wise dynamical properties (entry into a ridge tube, normal alignment, and tangent sliding) that shape generation. By decomposing training-induced errors into normal and tangent components and applying this to a random-feature neural network (RFNN) formalism, the paper links architecture and optimization to generation biases, yielding concrete predictions on when inter-mode generation arises. Empirical validation on synthetic 2D distributions and MNIST latent spaces supports the theory, showing that normalization toward ridges dominates early inference and tangential sliding governs the spread along ridges, with weight schedules and initialization modulating these effects.

Abstract

When a diffusion model is not memorizing the training data set, how does it generalize exactly? A quantitative understanding of the distribution it generates would be beneficial to, for example, an assessment of the model's performance for downstream applications. We thus explicitly characterize what diffusion model generates, by proposing a log-density ridge manifold and quantifying how the generated data relate to this manifold as inference dynamics progresses. More precisely, inference undergoes a reach-align-slide process centered around the ridge manifold: trajectories first reach a neighborhood of the manifold, then align as being pushed toward or away from the manifold in normal directions, and finally slide along the manifold in tangent directions. Within the scope of this general behavior, different training errors will lead to different normal and tangent motions, which can be quantified, and these detailed motions characterize when inter-mode generations emerge. More detailed understanding of training dynamics will lead to more accurate quantification of the generation inductive bias, and an example of random feature model will be considered, for which we can explicitly illustrate how diffusion model's inductive biases originate as a composition of architectural bias and training accuracy, and how they evolve with the inference dynamics. Experiments on synthetic multimodal distributions and MNIST latent diffusion support the predicted directional effects, in both low- and high-dimensions.
Paper Structure (32 sections, 25 theorems, 180 equations, 22 figures)

This paper contains 32 sections, 25 theorems, 180 equations, 22 figures.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Geometric Dynamical Properties of the Inference Process
  4. Data-dependent Manifolds - Log-density Ridge Sets
  5. Stage 1 - Reaching the Tube Neighborhood
  6. Stage 2 - Aligning along Normal Directions
  7. Stage 3 - Sliding along Tangent Directions
  8. How Training Affects Generation: General Theory + an Explicit RF Example
  9. Directional Decomposition along Log-density Ridges
  10. Application in RF+GD
  11. RFNN for Two Points in 2D
  12. Experiments
  13. Synthetic Data - Two Points in 2D Plane
  14. More Points in 2D
  15. Higher Dimension Data - MNIST
  16. ...and 17 more sections

Key Result

Proposition 3.1

Under Assumption assump:data, the log-density ridge sets $\{\mathcal{R}_t\}_{\delta\le t\le T}$ satisfy Assumption assump:ridge set with $r_t = \Omega(h_t^{2} \theta_t^{-1} R^{-3})$ and arbitrary $\theta_t=\exp(- o(h_{t}^{-1}) )$ as $t\to \delta^+\ll 1$. Furthermore, for any $\rho_t\in (0,r_t)$ and the nearest-point projection $\Pi_t: \mathcal{T}_t(\rho_t)\to \mathcal{R}_t$ is well-defined with t

Figures (22)

  • Figure 1: Illustration of the generation inductive bias. 13 training points (red crosses) form a letter 'M' in 2D. Generated samples (blue dots) evolve relative to the time-indexed log-density ridges (green curve) by following reach--align--slide. The background color represents the KDE.
  • Figure 2: Error dynamics and generated samples of RFNN. (a) Evolution of tangential errors (solid lines, left axis, linear scale) and normal errors (dash-dot lines, right axis, log scale) during training. (b)--(d) Comparison of generated sample configurations under different weighting schedules. Boxed numbers indicate sample counts around the target modes ($\text{radius}=0.5$). The background color represents the KDE plot.
  • Figure 3: Error dynamics and generated samples of MLP. (a) Evolution of tangential errors (solid lines, left axis, linear scale) and normal errors (dash-dot lines, right axis, log scale) during training. (b)--(d) Comparison of generated sample configurations under different weighting schedules. Boxed numbers indicate sample counts around the target modes ($\text{radius}=0.5$). The background color represents the KDE plot.
  • Figure 4: Loss Decomposition of RFNN. Evolution of tangent and normal loss components ($w(t)=1, h_t, h_t^2$) for the RFNN.
  • Figure 5: Initialization Effects (Epoch 40k). Comparison of generated samples under (a) zero (b) all-ones and (c) slow-spectrum initializations. The colored shading denotes the KDE of the distribution.
  • ...and 17 more figures

Theorems & Definitions (53)

  • Definition 3.1: Log-density Ridge Sets
  • Proposition 3.1
  • Theorem 3.1: Informal, formal one in Theorem \ref{['thm:formal stage 1']}
  • Theorem 3.2
  • Theorem 3.3
  • Remark 3.1: Comparison between normal&tangent directions
  • Theorem 4.1
  • Remark 4.1
  • Theorem 4.2: Training-time and width control geometric behavior
  • Proposition B.1
  • ...and 43 more