Understanding Generalizability of Diffusion Models Requires Rethinking the Hidden Gaussian Structure

TL;DR

Diffusion models trained on finite data generalize in part by learning a hidden Gaussian structure in their score functions. The authors show that, in the generalization regime, nonlinear diffusion denoisers become approximately linear and align with the optimal Gaussian denoiser for a data distribution characterized by mean $\boldsymbol{\mu}$ and covariance $\boldsymbol{\Sigma}$ estimated from training data. This Gaussian inductive bias is strongest when model capacity is small relative to data and can emerge early in training for overparameterized models, offering a potential explanation for observed strong generalization. The work provides a new perspective on diffusion model generalization by linking denoising behavior to data covariance and highlights the role of linear Gaussian structure alongside nonlinearity in high-quality image generation.

Abstract

In this work, we study the generalizability of diffusion models by looking into the hidden properties of the learned score functions, which are essentially a series of deep denoisers trained on various noise levels. We observe that as diffusion models transition from memorization to generalization, their corresponding nonlinear diffusion denoisers exhibit increasing linearity. This discovery leads us to investigate the linear counterparts of the nonlinear diffusion models, which are a series of linear models trained to match the function mappings of the nonlinear diffusion denoisers. Surprisingly, these linear denoisers are approximately the optimal denoisers for a multivariate Gaussian distribution characterized by the empirical mean and covariance of the training dataset. This finding implies that diffusion models have the inductive bias towards capturing and utilizing the Gaussian structure (covariance information) of the training dataset for data generation. We empirically demonstrate that this inductive bias is a unique property of diffusion models in the generalization regime, which becomes increasingly evident when the model's capacity is relatively small compared to the training dataset size. In the case that the model is highly overparameterized, this inductive bias emerges during the initial training phases before the model fully memorizes its training data. Our study provides crucial insights into understanding the notable strong generalization phenomenon recently observed in real-world diffusion models.

Figures (30)

• Figure 1: Linearity scores of diffusion denoisers. Solid and dashed lines depict the linearity scores across noise variances for models in the generalization and memorization regimes, respectively, where $\alpha=\beta=1/\sqrt{2}$.
• Figure 2: Score field approximation error and sampling Trajectory. The left and right figures demonstrate the score field approximation error and the sampling trajectories $\mathcal{D}(\bm x_t;\sigma(t)$ of actual diffusion model (EDM), Multi-Delta model, linear model and Gaussian model respectively. Notice that the curve corresponding to the Gaussian model almost overlaps with that of the linear model, suggesting they share similar funciton mappings.
• Figure 3: Images sampled from various Models. The figure shows the samples generated using different models starting from the same initial noises.
• Figure 4: Linear model shares similar function mapping with Gaussian model. The left figure shows the difference between the linear weights and the Gaussian weights w.r.t. 100 training epochs of the linear distillation process for the 10 discrete noise levels. The right figure shows the correlation matrices between the first 100 singular vectors of the linear weights and Gaussian weights.
• Figure 5: Comparison between the diffusion denoisers in memorization and generalization regimes. Figure(a) demonstrates that in the memorization regime (trained on small datasets of size 1094 and 68), $\mathcal{D}_{\mathrm{L}}$ significantly diverges from $\mathcal{D}_{\mathrm{G}}$, and both provide substantially poorer approximations of $\mathcal{D}_{\bm \theta}$ compared to the generalization regime (trained on larger datasets of size 35000 and 1094). Figure(b) qualitatively shows that the denoising outputs of $\mathcal{D}_{\bm \theta}$ closely match those of $\mathcal{D}_{\mathrm{G}}$ only in the generalization regime—a similarity that persists even when the denoisers process pure noise inputs.

Theorems & Definitions (1)

• Theorem 1