Critical Points of Random Neural Networks

TL;DR

This work provides a rigorous geometric analysis of the landscape of random neural networks at initialization in the infinite-width limit by modeling the network as an isotropic Gaussian field on the sphere and applying Kac–Rice methods. A key contribution is the identification of a trichotomy governed by the spectral derivative $\kappa'(1)$ of the covariance: for CRI$>2$ the expected number of critical points exhibits bounded, polynomial, or exponential growth in depth, with precise asymptotics given by constants and a depth-parametrized GOI framework; for irregular activations like ReLU (CRI$<2$) the classical formulas fail and numerical evidence suggests potential divergence of critical points with resolution. The results extend to counts above thresholds via tail integrals, and numerical experiments corroborate the theoretical regimes while highlighting the exceptional behavior of irregular activations. Overall, the paper provides a deep link between covariance regularity, spectral parameters, and the geometric complexity of neural-network-inspired random fields, offering insights into initialization-induced landscape structure.

Abstract

This work investigates the expected number of critical points of random neural networks with different activation functions as the depth increases in the infinite-width limit. Under suitable regularity conditions, we derive precise asymptotic formulas for the expected number of critical points of fixed index and those exceeding a given threshold. Our analysis reveals three distinct regimes depending on the value of the first derivative of the covariance evaluated at 1: the expected number of critical points may converge, grow polynomially, or grow exponentially with depth. The theoretical predictions are supported by numerical experiments. Moreover, we provide numerical evidence suggesting that, when the regularity condition is not satisfied (e.g. for neural networks with ReLU as activation function), the number of critical points increases as the map resolution increases, indicating a potential divergence in the number of critical points.

Figures (4)

• Figure 1: Number of minima and maxima points for the Gaussian activation functions with different values of $a$. The dashed lines represent our theoretical findings. These numbers are computed using $1000$ Monte Carlo replicas.
• Figure 2: Approximation of the $A_{0}$ as the number of Monte Carlo replicates increases for dimension $d=2$.
• Figure 3: Percentage of the variance explained by using the first $1536$ frequencies for random neural networks on $\mathbb S^2$ with $\sigma_3$ as activation function for different values of depth $L$. To obtain the plot, we compute the angular power spectrum of this network using a Gauss--Legendre quadrature with $5000$ points.
• Figure 4: Mean number of local maxima over $1000$ Monte Carlo replicates. The fields are shallow (one hidden layer) random neural networks with $1000$ neurons. We use three activation functions: Gaussian with $a=1+\sqrt 2$ (diamond), ReLU (square), and $\tanh$ (triangle).

Theorems & Definitions (20)

• Definition 3.1
• Remark 3.2
• Proposition 3.3
• Theorem 3.4
• Theorem 3.5
• Remark 3.6
• Proposition 4.1: Theorem 4.4 cheng2018expected
• Remark 4.2
• Lemma 4.3
• proof