Generalization through variance: how noise shapes inductive biases in diffusion models

TL;DR

The paper addresses why diffusion models generalize beyond their training data by showing that the denoising score matching objective optimizes a noisy proxy score, whose ensemble covariance—the V-kernel—drives the learned distribution away from exact memorization and toward boundary-filled generalization. Using a Martin-Siggia-Rose path-integral (MSR) framework, it derives an effective stochastic dynamics for typical learned samples and shows that generalization emerges when the V-kernel is nonzero. It analyzes two tractable regimes—expressive linear models and lazy infinite-width networks—to reveal how feature-related inductive biases interact with proxy-score covariance to shape gap filling and interpolation behavior. The work highlights that the forward process, training noise scale, and data layout jointly determine generalization patterns, offering a principled lens to understand when diffusion models memorize versus generalize, and under what architectural or data conditions benign or harmful generalization occurs.

Abstract

How diffusion models generalize beyond their training set is not known, and is somewhat mysterious given two facts: the optimum of the denoising score matching (DSM) objective usually used to train diffusion models is the score function of the training distribution; and the networks usually used to learn the score function are expressive enough to learn this score to high accuracy. We claim that a certain feature of the DSM objective -- the fact that its target is not the training distribution's score, but a noisy quantity only equal to it in expectation -- strongly impacts whether and to what extent diffusion models generalize. In this paper, we develop a mathematical theory that partly explains this 'generalization through variance' phenomenon. Our theoretical analysis exploits a physics-inspired path integral approach to compute the distributions typically learned by a few paradigmatic under- and overparameterized diffusion models. We find that the distributions diffusion models effectively learn to sample from resemble their training distributions, but with 'gaps' filled in, and that this inductive bias is due to the covariance structure of the noisy target used during training. We also characterize how this inductive bias interacts with feature-related inductive biases.

Figures (3)

• Figure 1: Visualization of proxy score variance ($\text{tr}(\bm{C})/[\text{tr}(\bm{C}) + \Vert \bm{s} \Vert_2^2]$) for four example 2D data distributions. Each example data distribution is supported on a small number of point masses (red dots). As $t$ changes (left: small $t$, right: large $t$), boundary regions at different scales are emphasized.
• Figure 2: Average learned distribution ($N = 100$) for a linear model with Gaussian features trained on different sample draws from a 1D data distribution $\{ -1, 0, 1\}$. Red: average learned distribution; black: true distribution; gray: PF-ODE approximation of true distribution. Different values of the time cutoff $\epsilon$ and ratio $F/P$ are shown. Note that there is more generalization as both become larger.
• Figure 3: Generalization of a 2D data distribution depends on features used and data orientation. Heatmaps of samples from $N = 100$ linear models are shown in different conditions, with training data (red dots) overlaid. Notice that which gaps are 'filled in', e.g., whether a square shape or cross shape is made, depends on both factors.

Theorems & Definitions (4)

• Proposition 3.1: Effective SDE description of typical learned distribution
• Proposition 4.1: Naive score estimator generalizes
• Proposition 5.1: Expressive linear models asymptotically generalize
• Proposition 5.2: Lazy neural networks asymptotically generalize