On the Anisotropy of Score-Based Generative Models

TL;DR

The paper addresses the lack of a unified theory for directional biases in score based generative models by introducing Score Anisotropy Directions (SADs), an architecture dependent basis capturing how networks preferentially learn along output-space directions. By defining the average geometry $\mathbf{G}_{\mathcal{F}}(\mathcal{P},\Theta)$ and hypothesizing that its eigenvectors are the SADs, the authors provide both analytical and empirical support showing how initialization geometry constrains generalization across diffusion architectures, including CNNs and transformers. The work demonstrates that data misalignment with the architecture induced geometry can improve downstream Wasserstein-based performance and that SADs adapt to architectural details, offering a principled, pre-training predictor of generalization. These insights link to existing theories on harmonic bases and geometric priors while providing a concrete, actionable framework for understanding and engineering diffusion models. The findings have implications for designing more reliable and efficient diffusion-based generative systems and for broader considerations of inductive biases in generative modeling.

Abstract

We investigate the role of network architecture in shaping the inductive biases of modern score-based generative models. To this end, we introduce the Score Anisotropy Directions (SADs), architecture-dependent directions that reveal how different networks preferentially capture data structure. Our analysis shows that SADs form adaptive bases aligned with the architecture's output geometry, providing a principled way to predict generalization ability in score models prior to training. Through both synthetic data and standard image benchmarks, we demonstrate that SADs reliably capture fine-grained model behavior and correlate with downstream performance, as measured by Wasserstein metrics. Our work offers a new lens for explaining and predicting directional biases of generative models.

Figures (9)

• Figure 1: Sphere modeling in subspaces of $\mathbb{R}^D$ ($D=256$) via DiT dit. The only difference is the choice of subspace: the left "sphere" lies in a subspace aligned with the network’s geometry, $\mathbf{G}_{\mathcal{F}}$, while the right is in a non-aligned subspace. Despite identical setups, their quality differs consistently across repeated trials, suggesting that alignment with architectural geometry controls generalization. We formalize these ideas in Sections \ref{['subsec:generalizationrankone']} and \ref{['subsec:generalizationgeneral']}.
• Figure 2: $\text{MSW}_2$ distance (computed over 10k test samples and 16384 projections) of iDDPM U-Net improveddiffusion architecture. Each pixel corresponds to a rank-one dataset of 16$\times$16 images (with 10k training samples) that is aligned with a basis element of the canonical basis, DCT, DST, (ordered) Hadamard transform or Haar wavelet transform. That is, for a given location (canonical) or frequency / sequency (DCT, DST, Hadamard) or scale, channel and location (Haar), we visualize the performance on the corresponding dataset. For ease of visualization, in the case of DCT, DST and Hadamard, we center the zero frequency dataset and extend the images to the left and top regions while respecting the symmetries of the transforms. See Appendix \ref{['appendix:setup']} for details.
• Figure 3: Responses of the iDDPM architecture improveddiffusion with a symmetrical initialization scheme. We show the default implementation, which uses nearest resampling layers, and a modified architecture that uses area resampling. We probe the models with a centered impulse input, shown on the left. Observe that the default resampling introduces asymmetry.
• Figure 4: Setting of Theorem \ref{['theorem:linearscore']} ($\sigma=1$) in $\mathbb{R}^5$ with SGD. We show error between $\boldsymbol{\Omega}_t=\boldsymbol{\Phi}\boldsymbol{\Theta}_t$ and the optimal operator, $\boldsymbol{\Omega}^{*}$, defined in Lemma \ref{['lemma:linearoptimalscore']}.
• Figure 5: Visualization of our argument for uncovering anisotropy directions in $\mathbb{R}^2$. We show contours of a hypothetical landscape, $\log p_{\boldsymbol{\theta},\sigma}(\boldsymbol{x}_{\sigma})$, where $\boldsymbol{v}_{\parallel}$ is parallel to the induced "manifold" and $\boldsymbol{v}_{\perp}$ is orthogonal.

Theorems & Definitions (17)

• Definition 1: Score Anisotropy Directions
• Theorem 1: DSM under anisotropy, proof in Appendix \ref{['appendix:linearscoreproof']}
• Definition 2: Average Geometry
• Conjecture 1
• Theorem 2: Extreme alignment, proof in Appendix \ref{['appendix:alignment']}
• Proposition 1: MLP geometry, proof in Appendix \ref{['appendix:mlpgeometryproof']}
• Proposition 2: CNN geometry, proof in Appendix \ref{['appendix:cnngeometryproof']}
• Proposition 3: Transformer geometry, proof in Appendix \ref{['appendix:transformergeometryproof']}
• Lemma 1
• proof