Optimal Initialization in Depth: Lyapunov Initialization and Limit Theorems for Deep Leaky ReLU Networks

TL;DR

This work analyzes initialization in deep, fixed-width Leaky ReLU networks by establishing a Law of Large Numbers and a Central Limit Theorem for the log-norm of activations, governed by a Lyapunov exponent $\lambda_{\mu,\phi}$. It derives closed-form Lyapunov expressions for Gaussian and orthogonal weight distributions, revealing that common initializations can yield negative exponents in low-width regimes, causing vanishing activations, while He- or orthogonal-based schemes may be stable only in the infinite-width limit. The authors propose Lyapunov Initialization (setting $\lambda_{\mu,\phi}=0$) and a Sampling variant to counter depth-dependent fluctuations, with explicit formulas for $\sigma_{\mathrm{crit}}$ and $\eta_{\mathrm{crit}}$. Empirical results on deep, narrow networks (polynomial regression and score learning) show faster convergence and better generalization than traditional initializations, validating the theoretical predictions. The work highlights a principled pathway to stabilization in deep networks and suggests a strong case for Leaky ReLU activations in depth-heavy, narrow architectures.

Abstract

The development of effective initialization methods requires an understanding of random neural networks. In this work, a rigorous probabilistic analysis of deep unbiased Leaky ReLU networks is provided. We prove a Law of Large Numbers and a Central Limit Theorem for the logarithm of the norm of network activations, establishing that, as the number of layers increases, their growth is governed by a parameter called the Lyapunov exponent. This parameter characterizes a sharp phase transition between vanishing and exploding activations, and we calculate the Lyapunov exponent explicitly for Gaussian or orthogonal weight matrices. Our results reveal that standard methods, such as He initialization or orthogonal initialization, do not guarantee activation stabilty for deep networks of low width. Based on these theoretical insights, we propose a novel initialization method, referred to as Lyapunov initialization, which sets the Lyapunov exponent to zero and thereby ensures that the neural network is as stable as possible, leading empirically to improved learning.

Figures (21)

• Figure 1: Visualization of the Law of Large Number for $\log |X_{\ell}|$ (Theorem \ref{['ExplicitLeakyReLULLN']}) in dimension $d = 2$ with a Leaky ReLU activation function $\phi(x) = \max(x,\alpha x)$ and $\alpha = 0.1$. We initialized the matrix coefficients with independent Gaussian coefficients chosen such that the Lyapunov exponent is zero and therefore $\frac{1}{\ell}\log |X_{\ell}|$ converges to zero almost surely.
• Figure 2: Visualization of the Central Limit Theorem (Theorem \ref{['ExplicitLeakyReLUCLT']}): Plot of $\frac{\log |X_{\ell}| - \ell \lambda_{\mu,\phi}}{\sqrt{\ell}}$ for $d = 2$, $\alpha = 0.1$ and $\ell = 1,2,3,4,5$ and $8$ initialized with Gaussian weights of mean zero and chosen such that the Lyapunov exponent $\lambda_{\mu,\phi} = 0$. We sampled 1,000,000 points and showed the histograms. The distribution already looks rather Gaussian after 4 layers.
• Figure 3: Faster and more accurate learning with Lyapunov-based initializations: While learning a 40 layer neural network of width 2, Lyapunov based methods converge faster. While Lyapunov Gaussian (dark blue) and Lyapunov Orthogonal (dark green) improve significantly over conventional methods in the short- and long-term with a plateau in the mid-term, their sampled versions (light blue and light green) are significantly better on every time scale. Note that Glorot is always beyond the range of the plot.
• Figure 4: Improved initial phase for learning a score: Both Sampled Lyapunov Gaussian (blue) and Sampled Lyapunov Orthogonal (green) show lower losses in the first $1,000$ steps while learning the score of a gaussian mixture.
• Figure 5: Visualization that Theorem \ref{['ExplicitLeakyReLUCLT']} fails for $\tanh$ networks: Plot of $\frac{\log |X_{\ell}|}{\sqrt{\ell}}$ for $d = 2$ and $\ell = 1,2,3,4,5$ and $8$ for $X_{\ell}$ a $\tanh$ network initialized with Gaussian weights of mean zero and unit variance. We sampled 1'000'000 points and showed the histograms. The distribution never looks Gaussian.

Theorems & Definitions (43)

• Theorem 3.3
• Theorem 3.4
• Lemma 3.5
• Theorem 4.1
• Theorem 4.2
• Theorem 4.3
• Definition 2.1
• Theorem 2.2
• Definition 2.3
• Theorem 2.4