Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gaussian Pre-Activations in Neural Networks: Myth or Reality?

Pierre Wolinski, Julyan Arbel

TL;DR

This work interrogates the common assumption that neural-network pre-activations are Gaussian at initialization, showing that finite-width networks often deviate with heavier tails. It introduces a principled framework to impose Gaussian pre-activations across depth by jointly designing weight distributions and activation functions, specifically proposing symmetric Weibull weight initialization $W hicksim ext{W}( heta,1)$ and two activation families $ ext{φ}_{ heta}^{ ext{o}}$ (odd) and $ ext{φ}_{ heta}^{ ext{p}}$ (positive). The paper derives constraints that ensure Gaussian propagation under a pre-activations independence assumption, develops exact (non-asymptotic) Edge of Chaos analyses, and provides extensive experiments on synthetic data and CIFAR-10 that demonstrate Gaussian pre-activations across layers and improved trainability for narrow/deep networks in some regimes. It also critically discusses Edge of Chaos limitations, contrasts with NTK-based analyses, and offers practical guidance and open-source code for practitioners to evaluate Gaussian pre-activations in their architectures. Overall, the work provides a principled, implementable path to realize and leverage Gaussian pre-activations in finite-width networks, while highlighting that Gaussianity is not universally guaranteed or always desirable outside of particular settings.

Abstract

The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. An assumption very commonly made in the field states that the pre-activations are Gaussian. Although this convenient Gaussian hypothesis can be justified when the number of neurons per layer tends to infinity, it is challenged by both theoretical and experimental works for finite-width neural networks. Our major contribution is to construct a family of pairs of activation functions and initialization distributions that ensure that the pre-activations remain Gaussian throughout the network's depth, even in narrow neural networks. In the process, we discover a set of constraints that a neural network should fulfill to ensure Gaussian pre-activations. Additionally, we provide a critical review of the claims of the Edge of Chaos line of works and build an exact Edge of Chaos analysis. We also propose a unified view on pre-activations propagation, encompassing the framework of several well-known initialization procedures. Finally, our work provides a principled framework for answering the much-debated question: is it desirable to initialize the training of a neural network whose pre-activations are ensured to be Gaussian? Our code is available on GitHub: https://github.com/p-wol/gaussian-preact/ .

Gaussian Pre-Activations in Neural Networks: Myth or Reality?

TL;DR

This work interrogates the common assumption that neural-network pre-activations are Gaussian at initialization, showing that finite-width networks often deviate with heavier tails. It introduces a principled framework to impose Gaussian pre-activations across depth by jointly designing weight distributions and activation functions, specifically proposing symmetric Weibull weight initialization and two activation families (odd) and (positive). The paper derives constraints that ensure Gaussian propagation under a pre-activations independence assumption, develops exact (non-asymptotic) Edge of Chaos analyses, and provides extensive experiments on synthetic data and CIFAR-10 that demonstrate Gaussian pre-activations across layers and improved trainability for narrow/deep networks in some regimes. It also critically discusses Edge of Chaos limitations, contrasts with NTK-based analyses, and offers practical guidance and open-source code for practitioners to evaluate Gaussian pre-activations in their architectures. Overall, the work provides a principled, implementable path to realize and leverage Gaussian pre-activations in finite-width networks, while highlighting that Gaussianity is not universally guaranteed or always desirable outside of particular settings.

Abstract

The study of feature propagation at initialization in neural networks lies at the root of numerous initialization designs. An assumption very commonly made in the field states that the pre-activations are Gaussian. Although this convenient Gaussian hypothesis can be justified when the number of neurons per layer tends to infinity, it is challenged by both theoretical and experimental works for finite-width neural networks. Our major contribution is to construct a family of pairs of activation functions and initialization distributions that ensure that the pre-activations remain Gaussian throughout the network's depth, even in narrow neural networks. In the process, we discover a set of constraints that a neural network should fulfill to ensure Gaussian pre-activations. Additionally, we provide a critical review of the claims of the Edge of Chaos line of works and build an exact Edge of Chaos analysis. We also propose a unified view on pre-activations propagation, encompassing the framework of several well-known initialization procedures. Finally, our work provides a principled framework for answering the much-debated question: is it desirable to initialize the training of a neural network whose pre-activations are ensured to be Gaussian? Our code is available on GitHub: https://github.com/p-wol/gaussian-preact/ .
Paper Structure (106 sections, 9 theorems, 90 equations, 19 figures, 2 tables)

This paper contains 106 sections, 9 theorems, 90 equations, 19 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Initialization strategies.
  3. Edge of Chaos (EOC).
  4. Bayesian prior and initialization distribution.
  5. Gaussian hypothesis for the pre-activations.
  6. Deviation from the Gaussian distribution in the finite-width case.
  7. Addressing the limitations of the EOC analysis.
  8. Motivation.
  9. Contributions.
  10. Summary of the paper.
  11. Common assumptions about pre-activations
  12. Neural networks as Gaussian processes.
  13. Assumption of independent pre-activations.
  14. Propagating pre-activations
  15. Propagation of the correlation between data points
  16. ...and 91 more sections

Key Result

Proposition 1

If $f(u^*) = u^*$ and $f$ is $\mathcal{C}^1$ in a neighborhood of $u^*$ with $f'(u^*) \in (-1, 1)$, then $u^*$ is a stable fixed point of $f$.

Figures (19)

  • Figure 1: Propagation of correlations $c^l_{ab}$ in a multilayer perceptron with activation function $\phi \in \{\mathrm{tanh}, \mathrm{ReLU}\}$ and inputs sampled from the CIFAR-10 dataset. According to the EOC claims, they should tend to 1 as $l \rightarrow \infty$. Each plot displays a $10 \times 10$ matrix $C_{pq}^l$ whose entries are the average correlation between the pre-activations propagated by samples from classes $p,q\in \{0, \cdots , 9\}$, at the input and right after layers $l \in \{10, 30, 50\}$. See Fig. \ref{['fig:corr_suppl']}, App. \ref{['app:expe:extra:MNIST']}, for results on MNIST.
  • Figure 2: Evolution of the $\mathcal{L}^{\infty}$-distance between the standard Gaussian $\mathcal{N}(0, 1)$ and: (solid line) the empirical CDF of the pre-activations; (dotted line) the the empirical CDF of the standardized pre-activations. This distance should remain close to zero if the pre-activations are Gaussian. The empirical CDF of $\mathcal{D}^l$ have been computed with $10000$ samples. The green dotted line corresponds to the threshold given by the Kolmogorov--Smirnov test with a $p$-value of $0.05$: any point above it corresponds to a distribution for which the Gaussian hypothesis should be rejected with a $p$-value of $0.05$.
  • Figure 3: Properties of the proposed counterexamples $\varphi_{\delta, \omega}$ represented in Fig. \ref{['fig:perlog:act']} for $\omega\in\{2,3,6\}$ and $\delta = 0.99$. Their variance map $\mathcal{V}$ has an infinite number of fixed points. In Fig. \ref{['fig:perlog:Vv']}, stable points are marked by crosses ($+$), and unstable points by bullets ($\bullet$), when they are away from $0$: two stable points appear for $\omega = 6$ (in red). As established in Proposition \ref{['prop:counter']}, $\sigma_w$ is tuned for every $\omega$ in such a way that $\mathcal{V}$ crosses the identity function an infinite number of times (not visible on the figure). In Fig. \ref{['fig:perlog:Vv_log']} (log-log scale), it is clearer that $\mathcal{V}$ has an infinite number of fixed points, due to regular oscillations (in log-log scale) below and above the identity. In Fig. \ref{['fig:perlog:Cc']}, we show that as $\omega$ grows, the correlation map $\mathcal{C}$ becomes closer to the identity function, which means that the correlation between data points tends to propagate perfectly. Note: since an infinite number of stable fixed points are available, we have arbitrarily picked one for each $\omega$, denoted by $v^*$. This choice does not affect the plot of the correlation map $\mathcal{C}$, due to the very specific structure of $\varphi_{\delta, \omega}$.
  • Figure 4: Building process of the initialization distributions $\mathrm{P}_l$ of the parameters $(\mathbf{W}^l, \mathbf{B}^l)$: (i) the pre-activations-related distribution $\mathcal{D}^l$ passes through a map $\mathcal{T}_l{[\mathrm{P}_l]}$ and becomes $\mathcal{D}^{l + 1}$; (ii) some statistical characteristic $\kappa^l$ of $\mathcal{D}^l$ can be computed with a function $\chi$: $\kappa^l = \chi(\mathcal{D}^l)$; (iii) we tune the $(\mathrm{P}_l)_l$ in order to make the sequence $(\kappa^l)_l$ constant.
  • Figure 5: How to build a random variable $W Y = W \phi_{\theta}^{\mathrm{o}/\mathrm{p}}(X) =: G \sim \mathcal{N}(0, 1)$, where $X \sim \mathcal{N}(0, 1)$? (a) Choose the distribution $\mathrm{P}_{\theta}$ of $W$, then (b) deduce the distribution $\mathrm{Q}_{\theta}$ of $Y$, and finally (c-d) find $\phi_{\theta}^{\mathrm{o}}$ and $\phi_{\theta}^{\mathrm{p}}$.
  • ...and 14 more figures

Theorems & Definitions (34)

  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Definition 1: Activation function $\varphi_{\delta, \omega}$
  • Definition 2: Stable fixed points
  • Proposition 1
  • Proposition 2
  • Remark 5
  • Remark 6
  • ...and 24 more