Exploring Loss Landscapes through the Lens of Spin Glass Theory

TL;DR

This work addresses the lack of a principled theory for generalization in overparameterized DNNs by forging a spin-glass-inspired lens on loss landscapes. It maps a one-hidden-layer ReLU network to an $H$-spin spherical spin-glass Hamiltonian under a sphere constraint, and develops four protocols—random walks in parameter space, interpolation between solutions, hierarchical clustering, and replica-symmetry-breaking-based interpretations—to probe landscape structure and solution organization. Empirically, it demonstrates multiple near-optimal minima, symmetry-induced copies via permutation in hidden layers, and RSB-like clustering, while showing that flatter minima, especially in small-batch training, correlate with better generalization. The approach provides a physics-informed framework for understanding training dynamics and generalization and suggests pathways to extend these insights to more complex architectures and training regimes, with potential impact on designing optimization strategies and regularization techniques.

Abstract

In the past decade, significant strides in deep learning have led to numerous groundbreaking applications. Despite these advancements, the understanding of the high generalizability of deep learning, especially in such an over-parametrized space, remains limited. For instance, in deep neural networks (DNNs), their internal representations, decision-making mechanism, absence of overfitting in an over-parametrized space, superior generalizability, etc., remain less understood. Successful applications are often considered as empirical rather than scientific achievement. This paper delves into the loss landscape of DNNs through the lens of spin glass in statistical physics, a system characterized by a complex energy landscape with numerous metastable states, as a novel perspective in understanding how DNNs work. We investigated the loss landscape of single hidden layer neural networks activated by Rectified Linear Unit (ReLU) function, and introduced several protocols to examine the analogy between DNNs and spin glass. Specifically, we used (1) random walk in the parameter space of DNNs to unravel the structures in their loss landscape; (2) a permutation-interpolation protocol to study the connection between copies of identical regions in the loss landscape due to the permutation symmetry in the hidden layers; (3) hierarchical clustering to reveal the hierarchy among trained solutions of DNNs, reminiscent of the so-called Replica Symmetry Breaking (RSB) phenomenon (i.e. the Parisi solution) in spin glass; (4) finally, we examine the relationship between the ruggedness of DNN's loss landscape and its generalizability, showing an improvement of flattened minima.

Figures (11)

• Figure 1: Different degrees of Replica Symmetry Breaking (RSB). The middle one is first-order RSB and the right one is the second-order RSB.
• Figure 2: Examples of random walks in the parameter space from solutions $\vec{\textbf{w}}_{trained}$ of $FC_1-512$ trained on MNIST dataset by TensorFlow Adam optimizer. The parameter changes $\Delta\vec{\textbf{w}}$ are drawn from a distribution $P(\Delta\vec{\textbf{w}})=\mathcal{N}(0, \sigma^2)$, with $\sigma$ shown in the legend; the original trained parameter configuration has a magnitude $\vert\vec{\textbf{w}}_{trained}\vert\approx 0.1$.
• Figure 3: The test loss on the line interpolated between the original configuration and the one with swapped neuron in the hidden layer. $M$ denotes the number of nodes in the middle layer. For the output layer, the number of nodes is $10$. The activation functions are ReLU and Softmax. Results are averaged over $50$ trials. The number of swaps $S$, varies as follows:$0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30$
• Figure 4: The distribution of Euclidean distance between solution pairs trained for a single hidden layer ReLU model with the Adam optimization algorithm by the MNIST dataset. The training is repeated to obtain $200$ trained solutions. The number before 'M', like '128M', denotes the number of nodes in the hidden layer.
• Figure 5: The dendrogram and the similarity matrix. (a) and (b) show dendrograms where leaf nodes represent trained solutions, and intersections indicate cluster formations. The horizontal arrangement reflects their similarity. In (c), (d), (e), and (f), the distance similarity matrices are reordered based on the dendrogram's sequence. Dark colors indicate high similarity (short distances), while light colors indicate low similarity (long distances).

Theorems & Definitions (5)

• Theorem 3.1
• Theorem A.1
• proof
• Theorem A.2
• proof