The Principles of Diffusion Models

TL;DR

The Principles of Diffusion Models presents a unified, principled framework for diffusion-based generative modeling by tying together variational, score-based, and flow-based viewpoints. It shows how forward noising processes and backward denoising dynamics can be understood through time-dependent velocity fields and the Fokker–Planck equation, enabling sampling via reverse-time SDEs or probability-flow ODEs. The text develops both discrete and continuous-time formalisms (DDPM, NCSN, Score SDE/PF-ODE) and explains how denoising, score matching, and diffusion are mathematically equivalent perspectives on the same underlying transport of probability mass. It further discusses practical aspects such as guidance, fast solvers, and diffusion-motivated flow-maps, framing diffusion models as a durable foundation for controllable, efficient generative modeling with rigorous theory. The work culminates in a cohesive view that positions diffusion as a continuum of stochastic dynamics from simple priors to complex data, with wide implications for speed, control, and deployment of generative systems.

Abstract

This monograph presents the core principles that have guided the development of diffusion models, tracing their origins and showing how diverse formulations arise from shared mathematical ideas. Diffusion modeling starts by defining a forward process that gradually corrupts data into noise, linking the data distribution to a simple prior through a continuum of intermediate distributions. The goal is to learn a reverse process that transforms noise back into data while recovering the same intermediates. We describe three complementary views. The variational view, inspired by variational autoencoders, sees diffusion as learning to remove noise step by step. The score-based view, rooted in energy-based modeling, learns the gradient of the evolving data distribution, indicating how to nudge samples toward more likely regions. The flow-based view, related to normalizing flows, treats generation as following a smooth path that moves samples from noise to data under a learned velocity field. These perspectives share a common backbone: a time-dependent velocity field whose flow transports a simple prior to the data. Sampling then amounts to solving a differential equation that evolves noise into data along a continuous trajectory. On this foundation, the monograph discusses guidance for controllable generation, efficient numerical solvers, and diffusion-motivated flow-map models that learn direct mappings between arbitrary times. It provides a conceptual and mathematically grounded understanding of diffusion models for readers with basic deep-learning knowledge.

Figures (58)

• Figure 1: Timeline of diffusion model perspectives. Each group shares the same color. In \ref{['ch:variational']}, Variational Autoencoder (VAE) kingma2013auto$\to$ Diffusion Probabilistic Models (DPM) sohl2015deep$\to$ DDPM ho2020denoising. In \ref{['ch:score-based', 'ch:score-sde']}, Energy-Based Model (EBM) ackley1985learning$\to$ Noise Conditional Score Network (NCSN) song2019generative$\to$ Score SDE song2020score. In \ref{['ch:flow-based']}, Normalizing Flow (NF) rezende2015variational$\to$ Neural ODE (NODE) chen2018neural$\to$ Flow Matching (FM) lipman2022flow.
• Figure 2: Part B. Unifying and Principled Perspectives on Diffusion Models. This diagram visually connects classical generative modeling approaches—Variational Autoencoders, Energy-Based Models, and Normalizing Flows—with their corresponding diffusion model formulations. Each vertical path illustrates a conceptual lineage, culminating in the continuous-time framework. The three views (Variational, Score-Based, and Flow-Based) offer distinct yet mathematically equivalent interpretations.
• Figure 3: Illustration of the target in DGM. Training a DGM is essentially minimizing the discrepancy between the model distribution $p_{\bm{\phi}}$ and the unknown data distribution $p_{\mathrm{data}}$. Since $p_{\mathrm{data}}$ is not directly accessible, this discrepancy must be estimated efficiently using a finite set of independent and identically distributed (i.i.d.) samples, ${\mathbf{x}}_i$, drawn from it.
• Figure 4: Computation graphs of prominent deep generative models. Top to bottom: EBM maps an input ${\mathbf{x}}$ to a scalar energy; AR generates a sequence $\{{\mathbf{x}}_\ell\}$ left to right with causal dependencies; VAE encodes ${\mathbf{x}}$ to a latent ${\mathbf{z}}$ and decodes to a reconstruction ${\mathbf{x}}'$; NF applies an invertible map ${\mathbf{f}}_{\bm{\phi}}$ between ${\mathbf{x}}$ and ${\mathbf{z}}$ and uses ${\mathbf{f}}_{\bm{\phi}}^{-1}$ to produce ${\mathbf{x}}'$; GAN transforms noise ${\mathbf{z}}$ to a sample ${\mathbf{x}}'$ that is judged against real ${\mathbf{x}}$ by a discriminator $D_{\bm\zeta}$; DM iteratively refines a noisy sample through a multi-step denoising chain $\{{\mathbf{x}}_\ell\}$. Boxes denote variables, trapezoids are learnable networks, ovals are scalars; arrows indicate computation flow.
• Figure 5: Illustration of a VAE. It consists of a stochastic encoder $q_{\bm{\theta}}({\mathbf{z}}|{\mathbf{x}})$ that maps data ${\mathbf{x}}$ to a latent variable ${\mathbf{z}}$, and a decoder $p_{\bm{\phi}}({\mathbf{x}}|{\mathbf{z}})$ that reconstructs data from the latent.

Theorems & Definitions (1)

• Lemma