Sinusoidal Initialization, Time for a New Start

TL;DR

Introduces Sinusoidal initialization as a deterministic alternative to random weight schemes, arguing that structure can improve convergence and stability from the first forward pass. Across CNNs, Vision Transformers, and language models, the method shows faster convergence and higher final accuracy in aggregate. The paper formalizes the Sinusoidal construct, provides a theoretical framework to explain reduced activation skewness, and validates the approach through extensive empirical benchmarking across diverse architectures, reporting average accuracy gains of ~4.9% and ~20.9% faster convergence. This work presents a compelling deterministic alternative to stochastic initialization with potential efficiency and reproducibility benefits for large-scale deep learning.

Abstract

Initialization plays a critical role in Deep Neural Network training, directly influencing convergence, stability, and generalization. Common approaches such as Glorot and He initializations rely on randomness, which can produce uneven weight distributions across layer connections. In this paper, we introduce the Sinusoidal initialization, a novel deterministic method that employs sinusoidal functions to construct structured weight matrices expressly to improve the spread and balance of weights throughout the network while simultaneously fostering a more uniform, well-conditioned distribution of neuron activation states from the very first forward pass. Because Sinusoidal initialization begins with weights and activations that are already evenly and efficiently utilized, it delivers consistently faster convergence, greater training stability, and higher final accuracy across a wide range of models, including convolutional neural networks, vision transformers, and large language models. On average, our experiments show an increase of 4.9% in final validation accuracy and 20.9% in convergence speed. By replacing randomness with structure, this initialization provides a stronger and more reliable foundation for Deep Learning systems.

Figures (11)

• Figure 1: Validation accuracy over training epochs for ResNet-50 on CIFAR-100, comparing Sinusoidal with other initialization schemes.
• Figure 2: Weight distribution under Sinusoidal initialization.
• Figure 3: Heatmap of the initialized weight matrix.
• Figure 4: Neuron activation states, in white active neurons ($>0$), for different initializations.
• Figure 5: Histogram of the statistic $S$ values, highlighting the link between large $|S|$ and neuron skewness at $\alpha = 0.3$.

Theorems & Definitions (10)

• Definition 1
• Theorem 1: Threshold Equivalence
• Theorem 2: Asymptotic skewness under random initialization
• Theorem 3: Row Cancellation
• Theorem 4: Threshold Equivalence, general version
• proof
• Theorem : [Theorem \ref{['thrm:corollary']} (Asymptotic skewness under random initialization)]
• proof
• Theorem : Theorem \ref{['thrm:s=0']} (Row Cancellation)
• proof