When Bias Helps Learning: Bridging Initial Prejudice and Trainability

TL;DR

The paper addresses how initialization statistics shape trainability and intrinsic biases in wide neural networks by linking mean-field (MF) theory with the initial guessing bias (IGB) framework. It proves an equivalence between MF and IGB, recasting the MF order/chaos boundary as a state of deep prejudice at initialization that is transient during learning, and identifies the edge of chaos (EOC) as the optimal trainability point. The authors demonstrate that the best initial conditions are not neutral but biased in a way that accelerates learning, with bias absorbed early in training, across MLPs, residual nets, and Vision Transformers. They provide extensive theoretical phase diagrams and empirical validations, including how architectural choices and pooling layers shift the EOC and the IGB dynamics, highlighting practical implications for model design and initialization strategies.

Abstract

Understanding the statistical properties of deep neural networks (DNNs) at initialization is crucial for elucidating both their trainability and the intrinsic architectural biases they encode prior to data exposure. Mean-field (MF) analyses have demonstrated that the parameter distribution in randomly initialized networks dictates whether gradients vanish or explode. Recent work has shown that untrained DNNs exhibit an initial-guessing bias (IGB), in which large regions of the input space are assigned to a single class. In this work, we provide a theoretical proof linking IGB to MF analyses, establishing that a network predisposition toward specific classes is intrinsically tied to the conditions for efficient learning. This connection leads to a counterintuitive conclusion: the initialization that optimizes trainability is systematically biased rather than neutral. We validate our theory through experiments across multiple architectures and datasets.

Figures (27)

• Figure 1: Example of pre-activation distributions for neutrality (left) and moderate prejudice (right) computed by sampling Gaussian variables with synthetic data. The inset plots show the distribution of the dataset elements classified into the reference class $G_{0}$. In the neutral phase, $G_{0}$ is centred around $0.5$, while with moderate prejudice, $G_{0}$ concentrates at the extremes. At the transition between these two phases, $G_{0}$ is uniformly distributed (middle).
• Figure 2: Convergence behaviour the correlation coefficient of ReLU and Tanh for a single MLP with width equal to 10 000 and depth 100. $\sigma^{2}_{b^{}}=0.1$ and $\sigma^{2}_{w^{}}$ varies uniformly from the ordered phase (blue) to the chaotic phase (red). The transition point is $\sigma^{2}_{w^{}}=2.0$ for ReLU and close to it for Tanh. Scatter points indicate the asymptotic values. The inset plots show the convergence rate for the correlation coefficient to ity asymptotic value $c_{}^{}$, always exponential for Tanh and power law for ReLU in the chaotic phase. Solid lines are computed using the IGB approach, while shaded areas represent the 90 $\%$ central confidence interval computed using the MF approach.
• Figure 3: Extensive phase diagrams of infinitely wide MLPs, where we can observe some phases described in Tab. \ref{['tab:phases_description']}. The EOC is indicated with a continuous red line and it becomes a single point for ReLU (unbounded). In general, red lines indicate the transition between vanishing/exploding gradients.
• Figure 4: Accuracies and max classification frequency for a Tanh MLPs trained on binarized fashion MNIST, and the large Vision Transformer fine-tuned on CIFAR100. The EOC corresponds to both the initial state with the fastest learning dynamics and maximally biased state. The green line corresponds to $1/n_c$, where $n_c$ is the number of classes.
• Figure 5: Global, favoured and unfavoured class train and test accuracies for a ReLU MLP trained on binarized fashion MNIST.

Theorems & Definitions (39)

• Definition 2.1: Activation Drift Ratio
• Theorem 3.1: Equivalence between MF and IGB, informal
• Proposition 4.1
• Lemma C.1
• proof
• Lemma C.2
• proof
• Definition D.1: Averages over the dataset
• Theorem D.2: IGB pre-activation distributions
• proof