Deep Neural Nets as Hamiltonians

TL;DR

The paper reframes neural networks at initialization as disordered Hamiltonians and applies the replica trick to compute disorder-averaged thermodynamic quantities, focusing on the energy landscape induced by random MLPs in the infinite-width limit. It derives a replicated action in terms of layer-wise overlap matrices and dual variables, then solves the resulting saddle-point equations using a zipper method, revealing a rich spectrum of thermodynamic structures from replica symmetry to full replica symmetry breaking depending on activation. For deep linear networks, the authors obtain exact RS solutions and corroborate them with random-matrix theory; for shaped activations they derive a depth-continuum description via ODEs that also favors RS at high depth. Numerically, Monte Carlo simulations validate the theoretical predictions, showing consistency between energy landscapes and the predicted overlap structures, and highlighting how certain activations (e.g., $\sin$) promote RSB while others remain RS. Overall, the work provides a novel thermodynamic lens on neural-network inputs, offering analytic and numerical tools to characterize extreme-input distributions and the geometry of high-dimensional energy landscapes.

Abstract

Neural networks are complex functions of both their inputs and parameters. Much prior work in deep learning theory analyzes the distribution of network outputs at a fixed a set of inputs (e.g. a training dataset) over random initializations of the network parameters. The purpose of this article is to consider the opposite situation: we view a randomly initialized Multi-Layer Perceptron (MLP) as a Hamiltonian over its inputs. For typical realizations of the network parameters, we study the properties of the energy landscape induced by this Hamiltonian, focusing on the structure of near-global minimum in the limit of infinite width. Specifically, we use the replica trick to perform an exact analytic calculation giving the entropy (log volume of space) at a given energy. We further derive saddle point equations that describe the overlaps between inputs sampled iid from the Gibbs distribution induced by the random MLP. For linear activations we solve these saddle point equations exactly. But we also solve them numerically for a variety of depths and activation functions, including $\tanh, \sin, \text{ReLU}$, and shaped non-linearities. We find even at infinite width a rich range of behaviors. For some non-linearities, such as $\sin$, for instance, we find that the landscapes of random MLPs exhibit full replica symmetry breaking, while shallow $\tanh$ and ReLU networks or deep shaped MLPs are instead replica symmetric.

Figures (5)

• Figure 1: Outputs for a random 1-hidden-layer neural network for two different activation functions. In all cases, the input is a 2-sphere in the input space, with radius scaling as $\sqrt N$. For $\sin$ activations, we see a complicated function with many similarly-deep local minima. For ReLU activation, we find only a single local minimum on the sphere.
• Figure 2: We study the distribution of overlaps $q$ between two inputs draws independently from the Gibbs distribution $\exp(-\beta H_{\text{tot}})$ coming from a one layer network with $\sin$ activations at infinite width. (Left) Graph of $q$ as a function of $\beta$ for a 50-step RSB ansatz. For $\beta$ below approximately $4.5$, every $q$ is the same because the system is replica symmetric. At larger $\beta$, however, the system exhibits what appears to be full RSB, with a standard bimodal $q$ distribution. (Right) Graph of $q$ as a function of $m$ for fixed $\beta=7.1$. It displays the standard Parisi behavior of having two constant sections connected by a non-constant piece.
• Figure 4: For $\psi=-0.2$, $c(t)$ decreases with depth. The final value of $c$ also decreases as $\beta^2$ increases. Since $c$ can be interpreted as the replica-to-replica variance in the position, we see that as the system gets colder the activations cluster more and more closely around the global minimum.
• Figure 5: The evolution of overlaps $q(t)$ and variation $c(t)$ in the Gibbs distribution $\exp(-\beta H_{\text{tot}})$ for deep shaped MLPs.
• Figure 6: In blue, the replica calculation of the energy at inverse temperature $\beta$. In orange, numerical results obtained by sampling 50 random MLPs of width 200 using stochastic gradient descent.