Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme

TL;DR

This work introduces an emergence-promoting initialization that leverages a multiscale, structure-based measure of emergence to boost neural network training dynamics. By scaling layer-wise variances after standard initialization as $\tilde{W}_{-n} = W_n/\alpha^n$ and $\tilde{W}_n = W_n\alpha^n$, the method increases the potential for emergent behavior without new optimization steps. Empirical results across CNNs for image recognition and transformers for machine translation show improved convergence and accuracy, particularly when batch normalization is used to enable larger scaling factors, and code is released for reproducibility. The work broadens initialization practice by linking nonlinear emergent properties to training dynamics, suggesting a practical path to more robust and capable models.

Abstract

Emergence in machine learning refers to the spontaneous appearance of complex behaviors or capabilities that arise from the scale and structure of training data and model architectures, despite not being explicitly programmed. We introduce a novel yet straightforward neural network initialization scheme that aims at achieving greater potential for emergence. Measuring emergence as a kind of structural nonlinearity, our method adjusts the layer-wise weight scaling factors to achieve higher emergence values. This enhancement is easy to implement, requiring no additional optimization steps for initialization compared to GradInit. We evaluate our approach across various architectures, including MLP and convolutional architectures for image recognition and transformers for machine translation. We demonstrate substantial improvements in both model accuracy and training speed, with and without batch normalization. The simplicity, theoretical innovation, and demonstrable empirical advantages of our method make it a potent enhancement to neural network initialization practices. These results suggest a promising direction for leveraging emergence to improve neural network training methodologies. Code is available at: https://github.com/johnnyjingzeli/EmergenceInit.

Figures (4)

• Figure 1: An illustration of emergence in the hierarchical system.
• Figure 2: (A) The increase of weight magnitude through the layers mimicks how the energy level through the pieces of a domino is increasing. (B) Comparison of training loss and emergence of Xavier, Kaiming and our initialization schemes. Emergence measure: Xavier: $5.03e^8$, Kaiming: $5.99e^8$, Ours: $10.87e^8$
• Figure 3: Training loss and test accuracy of MLP on CIFAR-10.
• Figure 4: Additional examples of quivers. (a): A quiver $Q$ with vertices $V = \{a,b,c,d\}$ and oriented edges $E = \{\alpha, \beta, \gamma, \delta\}$, (b) and (c): two quiver representations over $Q$. Adapted from derksen2017introduction.

Theorems & Definitions (2)

• proof
• proof