Computational Inertia as a Conserved Quantity in Frictionless and Damped Learning Dynamics

TL;DR

A conserved quantity in continuous-time optimization dynamics, termed computational inertia, is identified as the sum of kinetic energy and potential energy and remains invariant under idealized, frictionless training.

Abstract

We identify a conserved quantity in continuous-time optimization dynamics, termed computational inertia. Defined as the sum of kinetic energy (parameter velocity) and potential energy (loss), this scalar remains invariant under idealized, frictionless training. We formalize this conservation law, derive its analytic decay under damping and stochastic perturbations, and demonstrate its behavior in a synthetic system. The invariant offers a compact lens for interpreting learning trajectories, and may inform theoretical tools for analyzing convergence, stability, and training geometry.

Figures (5)

• Figure 1: Evolution of $\mathcal{I}(t)$ in the frictionless and damped regimes for $\mathcal{L}(w) = \frac{1}{2} w^2$. Energy is conserved in the idealized setting and dissipates under friction.
• Figure 2: Phase portraits of parameter dynamics for $\mathcal{L}(w) = \frac{1}{2}w^2$ under $\gamma=0$ (conservative) and $\gamma=0.4$ (dissipative). Energy conservation corresponds to closed orbits; damping induces spiral decay.
• Figure 3: Empirical relationship between damping coefficient $\gamma$ and average decay rate of $\mathcal{I}(t)$. Energy loss increases monotonically with damping strength.
• Figure 4: Trajectories on a 2D quadratic loss surface, colored by $\mathcal{I}(t)$. Conservative systems maintain energy along each path; dissipative systems show smooth color fading as energy decays.
• Figure 5: Discrete-time computational inertia $\mathcal{I}_t$ under $\eta = 0.01$ for $\mathcal{L}(w) = \frac{1}{2} w^2$. The quantity remains nearly conserved, validating the continuous-time approximation.