Emergent properties of the local geometry of neural loss landscapes

TL;DR

The paper tackles the puzzling local geometry of neural loss landscapes in classification by proposing a minimal random-gradient/Hessian model that simultaneously explains four observed properties: a bulk-plus-outlier Hessian spectrum, gradient confinement to the top Hessian subspace, non-monotonic growth of the top eigenvalue during training, and successful training within low-dimensional subspaces via a Goldilocks zone of higher curvature. By assuming weak logit curvature, clustering of logit gradients, and increasing logit variance (probability freezing), the authors derive a simple framework in which these phenomena arise, connect to random matrix theory (BBP transitions) and spin-glass concepts (Derrida’s REM), and demonstrate consistency with empirical data across architectures and datasets. The work provides a unifying, theory-driven explanation for emergent geometric properties of loss landscapes, with potential implications for optimization dynamics, generalization, and design of training regimes. It also highlights how broad theoretical tools can illuminate practical aspects of deep learning, suggesting avenues for extending the approach beyond standard classification tasks.

Abstract

The local geometry of high dimensional neural network loss landscapes can both challenge our cherished theoretical intuitions as well as dramatically impact the practical success of neural network training. Indeed recent works have observed 4 striking local properties of neural loss landscapes on classification tasks: (1) the landscape exhibits exactly $C$ directions of high positive curvature, where $C$ is the number of classes; (2) gradient directions are largely confined to this extremely low dimensional subspace of positive Hessian curvature, leaving the vast majority of directions in weight space unexplored; (3) gradient descent transiently explores intermediate regions of higher positive curvature before eventually finding flatter minima; (4) training can be successful even when confined to low dimensional {\it random} affine hyperplanes, as long as these hyperplanes intersect a Goldilocks zone of higher than average curvature. We develop a simple theoretical model of gradients and Hessians, justified by numerical experiments on architectures and datasets used in practice, that {\it simultaneously} accounts for all $4$ of these surprising and seemingly unrelated properties. Our unified model provides conceptual insights into the emergence of these properties and makes connections with diverse topics in neural networks, random matrix theory, and spin glasses, including the neural tangent kernel, BBP phase transitions, and Derrida's random energy model.

Figures (8)

• Figure 1: The experimental results on clustering of logit gradients for different datasets, architectures, non-linearities and stages of training. The green bars correspond to $q^{\mathrm{SLSC}}$ in Eq. \ref{['eq:q_SLSC']}, the red bars to $q^{\mathrm{SL}}$ in Eq. \ref{['eq:q_SL']}, and the blue bars to $q^{\mathrm{DL}}$ in Eq. \ref{['eq:q_DL']}. In general, the gradients with respect to weights of logits $k$ will cluster well regardless of the class of the datapoint $\mu$ they were evaluated at. For datapoints of true class $k$, they will cluster slightly better, while gradients of two logits $k \neq l$ will be nearly orthogonal. This is visualized in Fig \ref{['fig:logit_gradient_alignment_figure']}.
• Figure 2: A diagram of logit gradient clustering. The $k^\mathrm{th}$ logit gradients cluster based on $k$, regardless of the input datapoint $\mu$. The gradients coming from examples $\mu$ of the class $k$ cluster more tightly, while gradients of different logits $k$ and $l$ are nearly orthogonal.
• Figure 3: The motion of probabilities in the probability simplex a) during training in a real network, and b) as a function of logit variance $\sigma_z$ in our random model. (a) The distribution of softmax probabilities in the probability simplex for a 3-class subset of CIFAR-10 during an early, middle, and late stage of training a SmallCNN. (b) The motion of probabilities induced by increasing the logit variance $\sigma^2_z$ (blue to red) in our random model and the corresponding decrease in the entropy of the resulting distributions.
• Figure 4: The evolution of logit variance, logit gradient length, and weight space radius with training time. The top left panel shows that the logit variance across classes, averaged over examples, grows with training time. The top right panel shows that logit gradient lengths grow with training time. The bottom left panel shows the weight norm grows with training time. All 3 experiments were conducted with a SmallCNN on CIFAR-10. The bottom right panel shows the logit variance grows as one moves radially out in weight space, at random initialization, with no training involved, again in a SmallCNN.
• Figure 5: The Hessian eigenspectrum in our random model. Due to logit-clustering it exhibits a bulk + $C-1$ outliers. To obtain $C$ outliers, we can use mean logit gradients $\vec{c}_k$ whose lengths vary with $k$ (data not shown).