Dispelling the Curse of Singularities in Neural Network Optimizations

TL;DR

The paper identifies a curse of singularities in neural network optimization, where mutually reinforcing growth of parametric and representation singularities leads to training instability and sharp loss explosions. Through a combination of theory on a one-layer Transformer and empirical observations, it shows that gradient updates amplify dominant singular directions and that gradient norms become less constrained as singularity grows. To mitigate this, the authors propose Parametric Singularity Smoothing (PSS), a lightweight mechanism that detects instability via gradient norms and smooths the dominant spectrum of weight matrices, preserving learned directions and improving trainability. Extensive experiments across BERT, GPT-2, and larger models demonstrate that PSS expands the stable learning-rate range, reduces instability, and maintains or improves downstream performance with minimal computational overhead, offering a practical stabilization tool for large-scale training.

Abstract

This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization.

Figures (9)

• Figure 1: Evolution of $\operatorname{SR}(\mathbf{W})$, $\operatorname{SR}(\mathbf{Z})$, singularity alignment $\phi$, and combined training loss (blue) and gradient norm (green, log scale) in the feedforward module (FFN) across 12 layers. The top panel shows unstable training (LR=2e-4), with a loss spike and sharp SR drop near step 85800, linking singularities to instability; the bottom panel shows stable training (LR=1e-4).
• Figure 2: Final loss across LRs for each stabilization method (BERT-base left, GPT-2-Medium right).
• Figure 3: For BERT-base trained at a learning rate of 4e-4: (a) Test loss curves comparing the unstable naive baseline with PSS applied at various stages. (b) Test loss curves using different smoothing policies upon instability detection.
• Figure 4: Loss curves with different stabilization methods.
• Figure 5: Singular value distribution of the representation matrix in the embedding layer.

Theorems & Definitions (15)

• Definition 2.1: Parametric Singularity
• Definition 2.2: Representation Singularity
• Definition 2.3: Singularity Alignment
• Theorem 2.1: Amplification of Parametric Singularity
• Theorem 2.2: Amplification of Singularity Alignment
• Theorem 2.3: Amplification of Representation Singularity
• Theorem 2.4: Bounds on Gradient Frobenius Norm
• Theorem A.1: Amplification of Parametric Singularity
• proof
• Theorem A.2: Amplification of Singularity Alignment