Thermodynamic Performance Limits for Score-Based Diffusion Models

TL;DR

This work addresses the fundamental limits of score-based diffusion models from a thermodynamic perspective. It derives a thermodynamic lower bound on the negative log-likelihood, tying data entropy S0, equilibrium entropy S1, and the entropy-rate integral of the learned score, and it interprets the score network as a Maxwell's Demon operating within a stochastic thermodynamics framework. The main contributions include the bound NLL >= (S0+S1)/2 - (1/2) integral_0^1 dot S_theta(t) dt, the entropy-rate identities for controlled-forward processes, and empirical validation showing the predicted entropy-rate relations (dot S^e ≈ -2 dot S^i) and bound tightness on synthetic data. The results offer a physics-grounded view of generative modeling performance and point to practical implications for thermodynamic computing hardware and entropy-based diagnostics in diffusion modeling.

Abstract

We establish a fundamental connection between score-based diffusion models and non-equilibrium thermodynamics by deriving performance limits based on entropy rates. Our main theoretical contribution is a lower bound on the negative log-likelihood of the data that relates model performance to entropy rates of diffusion processes. We numerically validate this bound on a synthetic dataset and investigate its tightness. By building a bridge to entropy rates - system, intrinsic, and exchange entropy - we provide new insights into the thermodynamic operation of these models, drawing parallels to Maxwell's demon and implications for thermodynamic computing hardware. Our framework connects generative modeling performance to fundamental physical principles through stochastic thermodynamics.

Figures (1)

• Figure 1: Comparison between the NLL and theoretical lower bound across diffusion model configurations. (Left) NLL values versus the lower bound in Eq. (1) (dashed) for Gaussian and Uniform data under both variance exploding (VE) and variance preserving (VP) processes. Marker shape denotes data distribution and process, while color indicates the noise parameter $\sigma \in [10,30]$. (Right) Entropy rates (intrinsic $\dot S^i_{\boldsymbol \theta}$, exchange $\dot S^e_{\boldsymbol \theta}$, and system $\dot S_{\boldsymbol \theta}$) estimated from the score network yielding the best NLL in the Uniform (VE) case, confirming the predicted $2{:}1$ ratio $\dot S^e_{\boldsymbol \theta} = -2 \dot S^i_{\boldsymbol \theta}$.