A Random Matrix Theory Perspective on the Consistency of Diffusion Models

TL;DR

This work investigates why diffusion models trained on non-overlapping data splits produce consistent outputs for the same seed. It develops a random matrix theory (RMT) framework anchored in a linear Gaussian denoiser to quantify finite-sample effects, showing that data randomness renormalizes the noise scale via a self-consistent map $\sigma^{2}\to\kappa(\sigma^{2})$ and decomposes cross-split fluctuations into anisotropy, input inhomogeneity, and a global $1/n$ scaling. It extends deterministic-equivalence tools to fractional matrix powers to analyze full sampling trajectories and validates the theory on linear denoisers and deep networks, revealing an overshrinkage toward the dataset mean and structured inconsistencies across eigenmodes and inputs. The results offer a principled baseline for reproducibility in diffusion training by linking spectral data properties to the stability of generative outputs, and they suggest broad applicability to memory-generalization dynamics and noise-space design in diffusion and related models.

Abstract

Diffusion models trained on different, non-overlapping subsets of a dataset often produce strikingly similar outputs when given the same noise seed. We trace this consistency to a simple linear effect: the shared Gaussian statistics across splits already predict much of the generated images. To formalize this, we develop a random matrix theory (RMT) framework that quantifies how finite datasets shape the expectation and variance of the learned denoiser and sampling map in the linear setting. For expectations, sampling variability acts as a renormalization of the noise level through a self-consistent relation $σ^2 \mapsto κ(σ^2)$, explaining why limited data overshrink low-variance directions and pull samples toward the dataset mean. For fluctuations, our variance formulas reveal three key factors behind cross-split disagreement: \textit{anisotropy} across eigenmodes, \textit{inhomogeneity} across inputs, and overall scaling with dataset size. Extending deterministic-equivalence tools to fractional matrix powers further allows us to analyze entire sampling trajectories. The theory sharply predicts the behavior of linear diffusion models, and we validate its predictions on UNet and DiT architectures in their non-memorization regime, identifying where and how samples deviates across training data split. This provides a principled baseline for reproducibility in diffusion training, linking spectral properties of data to the stability of generative outputs.

Figures (53)

• Figure 1: Motivating observation and the linear theory. A. Diffusion models trained on non-overlapping data splits generate similar images from the same initial noise, even with different neural network architectures, consistent with results in kadkhodaie2024generalizationzhang2024emergence. Notably, generated samples from both splits are visually similar to the prediction from the Gaussian linear theory wang_unreasonable_2024. B. Quantification of A by paired image distances (MSE) averaging from 512 initial noises. The low-MSE block structure of the four DNNs and linear solution emphasize that this consistency effect is related to the linear structure. CNN1 denotes the CNN trained on split1, similar for CNN2, DiT1, DiT2; CNN1 nearest denotes the set of closest training set sample for the 512 generated image. We hide results for linear predictor of two splits since their samples are nearly identical with the linear predictor for the full dataset. Similar analysis for FFHQ64 is showed in Fig. \ref{['suppfig:motivation_schematics']}.
• Figure 2: Renormalization of noise and its effect on expectation of linear denoiser. A. Roadmap of our theory. B. The relationship between the renormalized and raw noise variance $\kappa(\sigma^2)$ as a function of $\gamma=d/n$, using the empirical spectrum of FFHQ32 as the limiting spectrum (plot underneath). See \ref{['app:numerical_methods']} for numerical methods. C. Shrinkage factor of linear denoiser along population eigenvectors at different noise scales. Empirical shows $\mathbf{v}^\top\hat{\bm{\Sigma}} (\hat{\bm{\Sigma}}+\sigma^2 I)^{-1} \mathbf{v}$ , when $\mathbf{v}=\mathbf{u}_k$ population PCs, at dataset size $n=1000,\gamma\approx3.1$. D. Schematics showing the overshrinking effect at lower eigenspaces, using linear denoiser outcome of faces as example.
• Figure 3: Structure of denoiser deviation across dataset splits.A. Visual examples of linear denoisers trained on two disjoint splits of FFHQ32 as noise variance $\sigma^2$ decreases, $n=1000$. Top, $\mathbf{x}_t$ noised sample; Bottom, output of linear denoiser (trained on split 1) $\mathbf D_{\hat{\bm{\Sigma}}_1}(\mathbf{x}_t,\sigma)$; Middle, deviation between two denoisers (normalized) $\mathbf D_{\hat{\bm{\Sigma}}_1}(\mathbf{x}_t,\sigma)-\mathbf D_{\hat{\bm{\Sigma}}_2}(\mathbf{x}_t,\sigma)$. At high noise, denoisers diverge on global, low-frequency content; at low noise, they deviate at specular details. B.Anisotropy: variance depends on probe direction $\mathbf v$; deviation is maximized when the eigenvalue $\lambda_k$ of $\mathbf v$ matches the renormalized noise $\kappa(\sigma^2)$, in agreement with theory. C.Inhomogeneity: variance depends on probe location $\mathbf x_t$; samples displaced along high-variance eigenmodes induce larger deviations. D.Global scaling: marginal deviation decays with dataset size $n$, vanishing in the infinite-sample limit.
• Figure 4: Finite sample effect on diffusion sampling map.A.Overshrinkage of expectation. Expected scaling along eigenmode of the empirical sampling map $\mathbf{u}_k^\top\hat{\bm{\Sigma}}^{1/2}\mathbf{u}_k$ compared to the ideal $\sqrt{\lambda_k}$, showing overshrinking along lower eigenmodes. B.Anisotropy of consistency. Cross-split MSE depends on probe direction $\mathbf{v}$, with larger deviation on top eigenspaces. C.Inhomogeneity of consistency. Cross-split MSE depends on input location $\bar{\mathbf{x}}$; samples displaced along high-variance modes exhibit larger disagreement. Colors denote dataset size, shared across A,B,C. D.Scaling of consistency by eigenband. Decomposition of MSE across eigenbands shows that lower-variance modes require substantially more samples before cross-split MSE decays. See also Fig. \ref{['suppfig:diff_sample_RMT_expectation_variance']}.
• Figure 5: DNN validation of theory.A. Samples generated by UNet (same two seeds) across training set sizes and splits (FFHQ64); similarity increases with $n$, and increasingly matches the population linear predictor (right). B. Nearest-neighbor MSE in training vs. control sets reveals memorization at small $n$, $n>3000$ shows no statistical difference between the splits. C. Overall consistency improve as a function of dataset size, with DiT more consistent than UNet at each $n$ (cross split MSE, mean$\pm$std). D. Variance of generated samples per eigenmode highlight insufficient variance (overshrinkage) in mid-to-low eigenmodes with limited dataset size. E. Cross-split MSE per eigenmode shows anisotropy of consistency (Fig. \ref{['fig:diff_sample_RMT_expectation_variance']}B). Further, per dataset size, deviation in top eigenmodes decrease the most. F. In the renormalization regime ($n=30$k), RMT predictions of seed-wise consistency correlate with empirical deviations.

Theorems & Definitions (27)

• Proposition 4.1: Deterministic equivalent of the denoiser expectation
• Proposition 4.2: Deterministic equivalent of the denoiser variance
• Proposition 5.1: Deterministic equivalence for expectation of diffusion sampling map
• Proposition 5.2: Deterministic equivalence for variance of diffusion sampling map
• Proposition 3.1: Main result, deterministic equivalence of the expectation of score and denoiser
• proof
• Proposition 3.2: Main result, deterministic equivalence of the denoiser variance
• proof
• proof
• Lemma 3.3: Expected MSE between two i.i.d. samples doubles the variance