The Loss Surfaces of Neural Networks with General Activation Functions

TL;DR

This work extends the spin‑glass analogy for neural network loss surfaces beyond ReLU activations by formulating a general piecewise linear activation model and analyzing it with supersymmetric random matrix theory. Using Kac‑Rice and GOE methods, the authors derive GOE expressions for the complexity of critical points and then perform a large‑N asymptotic analysis, revealing that the leading logarithmic complexity matches the ReLU case while the sharp complexity is modulated by a deterministic low‑rank perturbation from the activation. They establish a banded structure of low‑index critical points and provide exact leading terms in certain regimes (notably for very low energy), along with index‑preservation results under Hessian constraints. The conclusions indicate that the multi‑spin glass framework captures the coarse landscape geometry for general activations, while practical training dynamics may still depend on activation‑specific sharp features and data distributions. The methods developed—GOE with low‑rank perturbations and supersymmetric treatments—offer a toolkit for broader spin‑glass explorations in random neural‑network models and related high‑dimensional systems.

Abstract

The loss surfaces of deep neural networks have been the subject of several studies, theoretical and experimental, over the last few years. One strand of work considers the complexity, in the sense of local optima, of high dimensional random functions with the aim of informing how local optimisation methods may perform in such complicated settings. Prior work of Choromanska et al (2015) established a direct link between the training loss surfaces of deep multi-layer perceptron networks and spherical multi-spin glass models under some very strong assumptions on the network and its data. In this work, we test the validity of this approach by removing the undesirable restriction to ReLU activation functions. In doing so, we chart a new path through the spin glass complexity calculations using supersymmetric methods in Random Matrix Theory which may prove useful in other contexts. Our results shed new light on both the strengths and the weaknesses of spin glass models in this context.

Figures (11)

• Figure 1: Experimental distribution of $R_2$ (data averaging; each sample is a single neuron) for random MLP and LeNet ReLU networks, and i.i.d. normal and MNIST data. The blue line is a kernel density estimation fit.
• Figure 2: Experimental distribution of $\bar{R}_2$ (neuron averaging; each sample is a single datum) for random MLP and LeNet ReLU networks, and i.i.d. normal and MNIST data. The blue line is a kernel density estimation fit.
• Figure 3: Experimental distribution of $R_2$ (data averaging; each sample is a single neuron) for MLP and LeNet ReLU networks trained to high validation accuracy on MNIST, and evaluated on i.i.d. normal and MNIST data. The blue line is a kernel density estimation fit.
• Figure 4: Experimental distribution of $\bar{R}_2$ (neuron averaging; each sample is a single datum) for MLP and LeNet ReLU networks trained to high validation accuracy on MNIST, and evaluated on i.i.d. normal and MNIST data. The blue line is a kernel density estimation fit.
• Figure 5: Experimental distribution of $(R_2,R_3)$ (data averaging; each sample is a single neuron) for random MLP and LeNet HardTanh networks, and i.i.d. normal and MNIST data. The plots show 2d kernel density estimation fits of the joint and 1d fits of the marginals.

Theorems & Definitions (45)

• Remark 2.1
• Remark 2.2
• Definition 2.3
• Lemma 2.4
• proof
• Remark 2.5
• Theorem 2.6
• proof
• Lemma 2.7
• proof