Thermodynamically Optimal Regularization under Information-Geometric Constraints
Laurent Caraffa
TL;DR
This paper proposes a unifying framework linking thermodynamic optimality, information geometry, and regularization in machine learning. Under three explicit assumptions, it proves that Fisher--Rao geometry is the unique intrinsic metric on belief space and that thermodynamically optimal regularization corresponds to minimizing the squared Fisher--Rao distance to a reference state, with a Landauer-limited energy bound tied to the KL divergence between beliefs. It derives hyperbolic and von Mises geometric structures for Gaussian and circular beliefs, respectively, and argues that traditional Euclidean regularization is structurally suboptimal. The work introduces the thermodynamic efficiency of learning and a crystallization diagnostic, and it proposes three testable predictions and experimental protocols to validate the framework, potentially guiding energy-aware regularization in large-scale models.
Abstract
Modern machine learning relies on a collection of empirically successful but theoretically heterogeneous regularization techniques, such as weight decay, dropout, and exponential moving averages. At the same time, the rapidly increasing energetic cost of training large models raises the question of whether learning algorithms approach any fundamental efficiency bound. In this work, we propose a unifying theoretical framework connecting thermodynamic optimality, information geometry, and regularization. Under three explicit assumptions -- (A1) that optimality requires an intrinsic, parametrization-invariant measure of information, (A2) that belief states are modeled by maximum-entropy distributions under known constraints, and (A3) that optimal processes are quasi-static -- we prove a conditional optimality theorem. Specifically, the Fisher--Rao metric is the unique admissible geometry on belief space, and thermodynamically optimal regularization corresponds to minimizing squared Fisher--Rao distance to a reference state. We derive the induced geometries for Gaussian and circular belief models, yielding hyperbolic and von Mises manifolds, respectively, and show that classical regularization schemes are structurally incapable of guaranteeing thermodynamic optimality. We introduce a notion of thermodynamic efficiency of learning and propose experimentally testable predictions. This work provides a principled geometric and thermodynamic foundation for regularization in machine learning.