Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

Enrico Ventura, Beatrice Achilli, Gianluigi Silvestri, Carlo Lucibello, Luca Ambrogioni

TL;DR

The paper investigates the latent geometry of generative diffusion under the manifold hypothesis by analyzing the spectrum of the Jacobian of the score function. Employing random-matrix theory for linear-manifold data, it derives spectral distributions and gap formulas, then validates them with experiments on synthetic linear data and real image models. The authors identify three qualitative diffusion phases—trivial, manifold coverage, and manifold consolidation—and show how intermediate and final spectral gaps reveal subspace structure and manifold dimensionality, offering an explanation for why diffusion models avoid manifold overfitting. This framework connects the local geometry of diffusion trajectories to global data structure, providing a quantitative lens for understanding diffusion dynamics and guiding future analyses of latent manifolds in high-dimensional generative models.

Abstract

In this paper, we investigate the latent geometry of generative diffusion models under the manifold hypothesis. For this purpose, we analyze the spectrum of eigenvalues (and singular values) of the Jacobian of the score function, whose discontinuities (gaps) reveal the presence and dimensionality of distinct sub-manifolds. Using a statistical physics approach, we derive the spectral distributions and formulas for the spectral gaps under several distributional assumptions, and we compare these theoretical predictions with the spectra estimated from trained networks. Our analysis reveals the existence of three distinct qualitative phases during the generative process: a trivial phase; a manifold coverage phase where the diffusion process fits the distribution internal to the manifold; a consolidation phase where the score becomes orthogonal to the manifold and all particles are projected on the support of the data. This `division of labor' between different timescales provides an elegant explanation of why generative diffusion models are not affected by the manifold overfitting phenomenon that plagues likelihood-based models, since the internal distribution and the manifold geometry are produced at different time points during generation.

Manifolds, Random Matrices and Spectral Gaps: The geometric phases of generative diffusion

TL;DR

The paper investigates the latent geometry of generative diffusion under the manifold hypothesis by analyzing the spectrum of the Jacobian of the score function. Employing random-matrix theory for linear-manifold data, it derives spectral distributions and gap formulas, then validates them with experiments on synthetic linear data and real image models. The authors identify three qualitative diffusion phases—trivial, manifold coverage, and manifold consolidation—and show how intermediate and final spectral gaps reveal subspace structure and manifold dimensionality, offering an explanation for why diffusion models avoid manifold overfitting. This framework connects the local geometry of diffusion trajectories to global data structure, providing a quantitative lens for understanding diffusion dynamics and guiding future analyses of latent manifolds in high-dimensional generative models.

Abstract

In this paper, we investigate the latent geometry of generative diffusion models under the manifold hypothesis. For this purpose, we analyze the spectrum of eigenvalues (and singular values) of the Jacobian of the score function, whose discontinuities (gaps) reveal the presence and dimensionality of distinct sub-manifolds. Using a statistical physics approach, we derive the spectral distributions and formulas for the spectral gaps under several distributional assumptions, and we compare these theoretical predictions with the spectra estimated from trained networks. Our analysis reveals the existence of three distinct qualitative phases during the generative process: a trivial phase; a manifold coverage phase where the diffusion process fits the distribution internal to the manifold; a consolidation phase where the score becomes orthogonal to the manifold and all particles are projected on the support of the data. This `division of labor' between different timescales provides an elegant explanation of why generative diffusion models are not affected by the manifold overfitting phenomenon that plagues likelihood-based models, since the internal distribution and the manifold geometry are produced at different time points during generation.
Paper Structure (24 sections, 43 equations, 13 figures, 1 table, 2 algorithms)

This paper contains 24 sections, 43 equations, 13 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. The manifold hypothesis
  3. Background on generative diffusion models
  4. Dynamic latent manifolds and spectral gaps
  5. Subspaces and intermediate gaps
  6. Phenomenology of generative diffusion on manifolds
  7. Phase I: The trivial phase
  8. Phase II: Manifold coverage
  9. Phase III: Manifold consolidation
  10. The geometric phases and manifold overfitting
  11. Theoretical analysis of the spectral gaps in linear diffusion models
  12. Linear manifolds
  13. The isotropic case
  14. Intermediate gaps and subspaces with different variances
  15. Experiments with synthetic linear datasets
  16. ...and 9 more sections

Figures (13)

  • Figure 1: (a) Visualization of the gaps in the spectrum of the (negative) Jacobian of the score for data supported on a latent manifold. Blue line: idealized spectrum of distribution with uniform internal density; Orange line: spectrum of a more realistic distribution. (b) Sketch of the local structure of data-manifold with tangent and orthogonal components of the score function. (c) Sketch of the geometric phases of generative diffusion and their trace measurable from the eigenspectrum.
  • Figure 2: Spectrum of the eigenvalues of $J_{t}$ and drop in the dimensionality of the data-manifold estimated from theory in the double-variance case, with $\alpha_m = 0.5, \sigma_1^2 = 1,\sigma_2^2 = 0.01$, $f = 0.75$. Numerical data are generated with $d = 100$ and collected over $100$ realizations of the $F$ matrix.
  • Figure 3: Ordered singular values obtained with the trained score model, for different variances on the subspaces. Data are generated according to the linear-manifold model with $d = 100$ and $m = 40$. Left: $\sigma_1^2=1$, $\sigma_2^2=0.01$, $f = 0.75$; Center: $\sigma_1^2=0.01$, $\sigma_2^2=1$, $f = 0.75$; Right: a progressive number $n$ of variances sampled uniformly between $10^{-2}$ and $1$ each one assigned to a fraction $f = 1/n$ of matrix columns. The neural network is trained as prescribed in Supp. \ref{['supp: details training']} and spectra are measured according to Supp. \ref{['supp: details method']}.
  • Figure 4: Comparison between spectra obtained with the trained score model and with the numerical analysis, for different variances on the subspaces. Data are generated according to the linear-manifold model with $d = 100$ and $m = 40$, $\sigma_1^2=1$, $\sigma_2^2=0.01$, $f = 0.75$; from left to right, the spectra are evaluated a time $t\approx0.45$, $t\approx 0.3$, $t\approx 0.11$. The neural network is trained as prescribed in Supp. \ref{['supp: details training']} and spectra are measured according to Supp. \ref{['supp: details method']}.
  • Figure 5: Jacobian spectra of diffusion models trained on MNIST, Cifar10 and CelebA. The neural network is trained as prescribed in Supp. \ref{['supp: details training']} and spectra are measured according to Supp. \ref{['supp: details method']}.
  • ...and 8 more figures