An Introduction to Flow Matching and Diffusion Models

Abstract

Diffusion and flow-based models have become the state of the art for generative AI across a wide range of data modalities, including images, videos, shapes, molecules, music, and more. This tutorial provides a self-contained introduction to diffusion and flow-based generative models from first principles. We systematically develop the necessary mathematical background in ordinary and stochastic differential equations and derive the core algorithms of flow matching and denoising diffusion models. We then provide a step-by-step guide to building image and video generators, including training methods, guidance, and architectural design. This course is ideal for machine learning researchers who want to develop a principled understanding of the theory and practice of generative AI.

Figures (22)

• Figure 1: A flow $\psi_t:\mathbb R^d\rightarrow \mathbb R^d$ (red square grid) is defined by a velocity field $u_t :\mathbb R^d\rightarrow\mathbb R^d$ (visualized with blue arrows) that prescribes its instantaneous movements at all locations (here, $d=2$). We show three different times $t$. As one can see, a flow is a diffeomorphism that "warps" space. Figure from lipman2024flow.
• Figure 2: Sample trajectories of a Brownian motion $W_t$ in dimension $d=1$ simulated using \ref{['eq:brownian_motion_simulation']}.
• Figure 3: Illustration of Ornstein-Uhlenbeck processes (\ref{['eq:ohrstein_uhlenbeck_process']}) in dimension $d=1$ for $\theta=0.25$ and various choices of $\sigma$ (increasing from left to right). For $\sigma=0$, we recover a flow (smooth, deterministic trajectories) that converges to the origin as $t \to \infty$. For $\sigma>0$ we have random paths which converge towards the Gaussian $\mathcal{N}(0,\frac{\sigma^2}{2\theta})$ as $t\to\infty$.
• Figure 4: Gradual interpolation from noise to data via a Gaussian conditional probability path for a collection of images. Note that each image is a data point of dimension $d=32\times 32$, so we are plotting individual samples from the probability path, while in \ref{['fig:cond_marginal_path_histograms']} we plot the distribution as a 2d histogram.
• Figure 5: Illustration of a conditional (top) and marginal (bottom) probability path. Here, we plot a Gaussian probability path with $\alpha_t=t,\beta_t=1-t$. The conditional probability path interpolates a Gaussian $p_{\rm{init}}=\mathcal{N}(0,I_d)$ and $p_{\rm{data}}=\delta_{z}$ for single data point $z$. The marginal probability path interpolates a Gaussian and a data distribution $p_{\rm{data}}$ (Here, $p_{\rm{data}}$ is a toy distribution in dimension $d=2$ represented by a chess board pattern.)

Theorems & Definitions (54)

• Remark 1: Additional Resources
• Theorem 3: Flow existence and uniqueness
• Example 4: Linear Vector Fields
• Theorem 5: SDE Solution Existence and Uniqueness
• Example 6: Ornstein-Uhlenbeck Process
• Example 8: Gaussian Conditional Probability Path
• Theorem 9: Marginalization trick
• Example 10: Target ODE for Gaussian probability paths
• proof
• Theorem 11: Continuity Equation