Laws of thermodynamics for exponential families

TL;DR

The paper develops a thermodynamic framework for exponential-family learning, equating log-loss minimization with energy concepts and deriving analogues of the first and second laws under distribution shift. It shows that learning dynamics can be decomposed into work (reducible loss) and heat (irreducible loss), with the free energy capturing model regret and the entropy encapsulatingBayes risk. By linking the log-partition function, internal energy, and constraints through information projections, the authors derive Jarzynski-type inequalities, Le Chatelier principles, and fluctuation relations that illuminate how data perturbations and parameter shifts affect predictive performance. The framework unifies statistical physics and information theory, offering principled tools for understanding domain shift, robustness, and generalization in AI systems. Practical implications include principled analyses of model updates, distribution shift resilience, and energy-efficient learning strategies grounded in thermodynamic analogies.

Abstract

We develop the laws of thermodynamics in terms of general exponential families. By casting learning (log-loss minimization) problems in max-entropy and statistical mechanics terms, we translate thermodynamics results to learning scenarios. We extend the well-known way in which exponential families characterize thermodynamic and learning equilibria. Basic ideas of work and heat, and advanced concepts of thermodynamic cycles and equipartition of energy, find exact and useful counterparts in AI / statistics terms. These ideas have broad implications for quantifying and addressing distribution shift.