Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Fourier Space Perspective on Diffusion Models

Fabian Falck, Teodora Pandeva, Kiarash Zahirnia, Rachel Lawrence, Richard Turner, Edward Meeds, Javier Zazo, Sushrut Karmalkar

TL;DR

We analyze diffusion models in Fourier space to understand how the forward noising schedule interacts with data spectra and creates a high-frequency bias in DDPM. We introduce EqualSNR, a forward process that equalizes the per-frequency SNR, along with a training, sampling, and calibration procedure, and we also consider a FlippedSNR variant. Empirically, EqualSNR yields comparable imaging performance to DDPM on standard benchmarks while significantly improving high-frequency generation quality, and it excels on synthetic data where high-frequency details are paramount. The work provides a practical, spectrum-aware methodology for designing diffusion priors tailored to modality-specific Fourier properties, with implications for applications and safety.

Abstract

Diffusion models are state-of-the-art generative models on data modalities such as images, audio, proteins and materials. These modalities share the property of exponentially decaying variance and magnitude in the Fourier domain. Under the standard Denoising Diffusion Probabilistic Models (DDPM) forward process of additive white noise, this property results in high-frequency components being corrupted faster and earlier in terms of their Signal-to-Noise Ratio (SNR) than low-frequency ones. The reverse process then generates low-frequency information before high-frequency details. In this work, we study the inductive bias of the forward process of diffusion models in Fourier space. We theoretically analyse and empirically demonstrate that the faster noising of high-frequency components in DDPM results in violations of the normality assumption in the reverse process. Our experiments show that this leads to degraded generation quality of high-frequency components. We then study an alternate forward process in Fourier space which corrupts all frequencies at the same rate, removing the typical frequency hierarchy during generation, and demonstrate marked performance improvements on datasets where high frequencies are primary, while performing on par with DDPM on standard imaging benchmarks.

A Fourier Space Perspective on Diffusion Models

TL;DR

We analyze diffusion models in Fourier space to understand how the forward noising schedule interacts with data spectra and creates a high-frequency bias in DDPM. We introduce EqualSNR, a forward process that equalizes the per-frequency SNR, along with a training, sampling, and calibration procedure, and we also consider a FlippedSNR variant. Empirically, EqualSNR yields comparable imaging performance to DDPM on standard benchmarks while significantly improving high-frequency generation quality, and it excels on synthetic data where high-frequency details are paramount. The work provides a practical, spectrum-aware methodology for designing diffusion priors tailored to modality-specific Fourier properties, with implications for applications and safety.

Abstract

Diffusion models are state-of-the-art generative models on data modalities such as images, audio, proteins and materials. These modalities share the property of exponentially decaying variance and magnitude in the Fourier domain. Under the standard Denoising Diffusion Probabilistic Models (DDPM) forward process of additive white noise, this property results in high-frequency components being corrupted faster and earlier in terms of their Signal-to-Noise Ratio (SNR) than low-frequency ones. The reverse process then generates low-frequency information before high-frequency details. In this work, we study the inductive bias of the forward process of diffusion models in Fourier space. We theoretically analyse and empirically demonstrate that the faster noising of high-frequency components in DDPM results in violations of the normality assumption in the reverse process. Our experiments show that this leads to degraded generation quality of high-frequency components. We then study an alternate forward process in Fourier space which corrupts all frequencies at the same rate, removing the typical frequency hierarchy during generation, and demonstrate marked performance improvements on datasets where high frequencies are primary, while performing on par with DDPM on standard imaging benchmarks.
Paper Structure (48 sections, 4 theorems, 32 equations, 38 figures, 4 tables, 2 algorithms)

This paper contains 48 sections, 4 theorems, 32 equations, 38 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background: a spectral analysis of DDPM
  3. The DDPM forward process.
  4. DDPM corrupts high-frequencies faster and earlier.
  5. Signal-to-Noise Ratio.
  6. Fast noising disrupts high-frequency generation: alternate forward processes
  7. Faster noising of high frequencies violates the Gaussian assumption in their reverse process.
  8. Alternate forward processes
  9. Training algorithm and ELBO.
  10. Sampling algorithm.
  11. Calibration.
  12. Experiments
  13. Experiment setting.
  14. The reverse process learns to mirror its forward.
  15. Faster noising degrades high-frequency generation quality in DDPM, EqualSNR overcomes this.
  16. ...and 33 more sections

Key Result

Proposition 1

There is a choice of sufficiently small positive constants $\delta, \tau$ such that the following holds. Let $D_{0} = \tfrac{1}{2}\,\mathcal{N}(-1,\,\delta^2) + \tfrac{1}{2}\,\mathcal{N}(1,\,\delta^2)$; and $\mathbf{x}_{t-1} \sim D_{0}$ and let $\boldsymbol{\varepsilon}\sim \mathcal{N}(0,4)$. Then f

Figures (38)

  • Figure 1: [Left] The Fourier power law observed in (top-left) images krizhevsky2009learning, (top-right) videos kay2017kinetics, (bottom-left) audio gtanz1999music, and (bottom-right) Cryo-EM derived protein density maps wwpdb2024emdb. [Center] A DDPM forward process on these modalities noises high-frequency components substantially faster (SNR changes more per time increment), and earlier than low-frequency components. [Right] The alternate EqualSNR forward process noises all frequencies at the same rate, disrupting DDPM's generation hierarchy. The GIFs are best viewed in Adobe Reader.
  • Figure 3: Fast noising of high frequencies leads to violations of normality in the DDPM reverse process. We plot Monte Carlo estimates of $q(\mathbf{y}_t) = \mathbb{E}_{\mathbf{y}_0 \sim q(\mathbf{y}_0)} q(\mathbf{y}_t | \mathbf{y}_0)$ (and similarly for $q(\mathbf{y}_{t-1})$) as histograms.
  • Figure 4: The forward process controls when frequencies are generated in the reverse process. We visualise the forward and backward process of [left] DDPM and [right] EqualSNR in pixel space (rows 1,3,4) and Fourier space (magnitudes; row 2). Rows 3 and 4 are low- and high-pass filters of the (noisy) image. DDPM noises high-frequency components first and hence generates them last, while EqualSNR noises and generates all frequencies at the same time.
  • Figure 5: EqualSNR is superior to DDPM in high-frequency generation quality. We plot the spectral magnitude profile (in decibels) for low and high frequencies, comparing data generated with [top] DDPM and [bottom] EqualSNR generated (blue) and real (red) data.
  • Figure 6: EqualSNR outperforms DDPM on data where high-frequency information is dominant. Pixel intensity distribution (sorted descendingly) of 1000 generated samples for DDPM and EqualSNR, and two examples.
  • ...and 33 more figures

Theorems & Definitions (13)

  • Definition 1: Signal-to-Noise Ratio
  • Proposition 1: (Informal) Counterexample to normality of $q(\mathbf{y}_{t-1} | \mathbf{y}_{t})$
  • Definition 2: EqualSNR process in Fourier space
  • Definition 3: Modified forward process in Fourier space
  • Proposition 2: Loss Minimization as ELBO Maximization
  • proof : Proof Sketch
  • Definition 4: SNR
  • Definition 5: Multivariate SNR
  • Lemma 1: SNR and Covariance
  • proof
  • ...and 3 more