The Information Dynamics of Generative Diffusion

Abstract

Generative diffusion models have emerged as a powerful class of models in machine learning, yet a unified theoretical understanding of their operation is still developing. This paper provides an integrated perspective on generative diffusion by connecting the information-theoretic, dynamical, and thermodynamic aspects. We demonstrate that the rate of conditional entropy production during generation (i.e., the generative bandwidth) is directly governed by the expected divergence of the score function's vector field. This divergence, in turn, is linked to the branching of trajectories and generative bifurcations, which we characterize as symmetry-breaking phase transitions in the energy landscape. Beyond ensemble averages, we demonstrate that symmetry-breaking decisions are revealed by peaks in the variance of pathwise conditional entropy, capturing heterogeneity in how individual trajectories resolve uncertainty. Together, these results establish generative diffusion as a process of controlled, noise-induced symmetry breaking, in which the score function acts as a dynamic nonlinear filter that regulates both the rate and variability of information flow from noise to data.

Figures (5)

• Figure S1: Left: Fixed points of the score field. Right: Conditional entropy rate. The black dashed line denotes the stable fixed-point trajectories, while the red solid line represents the conditional entropy production rate. The background color indicates the logarithm of the process density.
• Figure S2: Stability and instability of trajectories in different parts of a symmetry-breaking potential. Generative branching is associated with divergent trajectories.
• Figure S3: An example of pathwise conditional entropies and corresponding paths for a four-point dataset located at $+2.3$, $+1.7$, $-1.7$, and $-2.3$. Each path is plotted in a distinct color, and the corresponding conditional-entropy curve uses the same color for visual matching. The background shading on the right panel depicts the normalized marginal density of the underlying SDE, with darker regions indicating lower probability.
• Figure S4: Evolution of the variance of the pathwise conditional entropy as a function of time for a one-dimensional mixture of two Gaussians with means $\pm \sqrt{\frac{4}{3} d}$ and variances $1$.
• Figure A1: Variance of the pathwise conditional entropy $h_t({\mathbf{x}}_t)$ against time $t$ for equiprobable two-component Gaussian mixtures for (a) EDM and (b) VP SDEs and several values of $d$. In EDM, the transition region broadens with $d$, whereas in VP, it remains approximately constant in width, consistent with a transition that localizes in (rescaled) time for VP but not for EDM.