Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

Luca Ambrogioni

TL;DR

This work reframes generative diffusion as an equilibrium statistical-mechanics problem, introducing Boltzmann ensembles over noiseless data conditioned on noisy states and deriving a self-consistent equation of state for the generative dynamics. It demonstrates that diffusion-based generation undergoes second-order phase transitions of mean-field universality, driven by spontaneous symmetry breaking and characterized by diverging susceptibility and canonical critical exponents. The authors extend the mean-field picture with a multi-site 'generative bath' to explore beyond-mean-field behavior, connect diffusion to associative memory/Hopfield networks, and analyze finite-sample effects through a random-energy REM lens, including memorization via condensation. Experimental evidence from trained diffusion models supports phase-transition–like behavior, including hierarchical feature emergence and sensitivity around critical times, with implications for sample diversity, memorization, and generation quality.

Abstract

Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.

The statistical thermodynamics of generative diffusion models: Phase transitions, symmetry breaking and critical instability

TL;DR

This work reframes generative diffusion as an equilibrium statistical-mechanics problem, introducing Boltzmann ensembles over noiseless data conditioned on noisy states and deriving a self-consistent equation of state for the generative dynamics. It demonstrates that diffusion-based generation undergoes second-order phase transitions of mean-field universality, driven by spontaneous symmetry breaking and characterized by diverging susceptibility and canonical critical exponents. The authors extend the mean-field picture with a multi-site 'generative bath' to explore beyond-mean-field behavior, connect diffusion to associative memory/Hopfield networks, and analyze finite-sample effects through a random-energy REM lens, including memorization via condensation. Experimental evidence from trained diffusion models supports phase-transition–like behavior, including hierarchical feature emergence and sensitivity around critical times, with implications for sample diversity, memorization, and generation quality.

Abstract

Generative diffusion models have achieved spectacular performance in many areas of machine learning and generative modeling. While the fundamental ideas behind these models come from non-equilibrium physics, variational inference and stochastic calculus, in this paper we show that many aspects of these models can be understood using the tools of equilibrium statistical mechanics. Using this reformulation, we show that generative diffusion models undergo second-order phase transitions corresponding to symmetry breaking phenomena. We show that these phase-transitions are always in a mean-field universality class, as they are the result of a self-consistency condition in the generative dynamics. We argue that the critical instability that arises from the phase transitions lies at the heart of their generative capabilities, which are characterized by a set of mean-field critical exponents. Finally, we show that the dynamic equation of the generative process can be interpreted as a stochastic adiabatic transformation that minimizes the free energy while keeping the system in thermal equilibrium.
Paper Structure (22 sections, 48 equations, 8 figures)

This paper contains 22 sections, 48 equations, 8 figures.

Table of Contents

  1. Introduction
  2. Contributions and Related Work
  3. Preliminaries on generative diffusion models
  4. Training diffusion models as denoising autoencoders
  5. Diffusion models as systems in equilibrium
  6. Example 1: Two deltas
  7. Example 2: Discrete dataset
  8. Example 3: Hyper-spherical manifold
  9. Example 4: Diffused Ising model
  10. Free energy, magnetization and order parameters
  11. The susceptibility matrix
  12. Phase transitions and symmetry breaking
  13. Generation and critical instability
  14. Generation as as an adiabatic free energy descent process
  15. Beyond mean-field theory: A multi-site 'generative bath' model
  16. ...and 7 more sections

Figures (8)

  • Figure 1: Generative process for a digit taken from the MNIST dataset.
  • Figure 2: Visualization of a phase transition in a simple diffusion model (two deltas). a) Order parameter paths and (regularized) free energy gradients. The dashed line denotes the critical value of $\sigma_t = \sqrt{t} \sigma_0$. b) Forward process. The dashed line denotes time critical time.
  • Figure 3: Negative free energy of a "four delta" 2d diffusion model for different values of the time variable. The target points are at $(1,0)$, $(0,1)$, $(-1,0)$ and $(0,-1)$.
  • Figure 4: MNIST
  • Figure 5: CIFAR10
  • ...and 3 more figures