Learning What Matters: Steering Diffusion via Spectrally Anisotropic Forward Noise

TL;DR

Diffusion probabilistic models rely on implicit inductive biases that shape learning. The authors introduce Spectrally Anisotropic Gaussian Diffusion (SAGD), which replaces the isotropic forward noise with a frequency-diagonal covariance in the Fourier domain, enabling targeted inductive biases via power-law or band-pass noise. They prove that, with full spectral support, the learned score converges to the true data score as $t\to0$, while anisotropy reshapes the learning trajectory. Empirically, SAGD improves image synthesis quality across diverse datasets, including large-scale ImageNet-1k in latent space, and enables selective omission of corrupted information by zeroing specific frequency bands. Overall, SAGD offers a simple, principled, drop-in mechanism to tailor inductive biases in diffusion-based generation.

Abstract

Diffusion Probabilistic Models (DPMs) have achieved strong generative performance, yet their inductive biases remain largely implicit. In this work, we aim to build inductive biases into the training and sampling of diffusion models to better accommodate the target distribution of the data to model. We introduce an anisotropic noise operator that shapes these biases by replacing the isotropic forward covariance with a structured, frequency-diagonal covariance. This operator unifies band-pass masks and power-law weightings, allowing us to emphasize or suppress designated frequency bands, while keeping the forward process Gaussian. We refer to this as Spectrally Anisotropic Gaussian Diffusion (SAGD). In this work, we derive the score relation for anisotropic forward covariances and show that, under full support, the learned score converges to the true data score as $t\!\to\!0$, while anisotropy reshapes the probability-flow path from noise to data. Empirically, we show the induced anisotropy outperforms standard diffusion across several vision datasets, and enables selective omission: learning while ignoring known corruptions confined to specific bands. Together, these results demonstrate that carefully designed anisotropic forward noise provides a simple, yet principled, handle to tailor inductive bias in DPMs.

Figures (7)

• Figure 1: Spectrally Anisotropic Gaussian Diffusion under a generalized framework.
• Figure 2: Power spectra and image visuals of the forward Process in standard diffusion, as compared to high ($\alpha=0.5$) and low-frequency ($\alpha=-0.5$) noise settings of a power-law weighted SAGD.
• Figure 4: Mean FID across seeds of $\textit{plw}\text{-SAGD}$ diffusion samplers trained on ImageNet1k ($\alpha=0$ yields standard diffusion).
• Figure 5: Samples from the original data distribution, the degraded data distribution, a standard diffusion sampler trained on the degraded data distribution, and a frequency diffusion sampler trained on the degraded data distribution. We generate noise for data corruption in the frequency range [$a_c=0.4$, $b_c=0.5)$].
• Figure D.1: Particle trajectories under the probability–flow ODE from a Gaussian prior to a mixture-of-Gaussians target (black contours), visualized at five equally spaced times (left to right). Rows: (top) isotropic noise ($\alpha{=}0$), (middle) high-frequency tilt ($\alpha{=}0.1$), (bottom) low-frequency tilt ($\alpha{=}{-}0.1$). $\textit{plw}\text{-SAGD}$ alters the path by reweighting modes via $\Sigma_w$ while keeping the endpoint consistent under full support (cf. Sec. \ref{['sec:analysis']}).