On the dynamics of three-layer neural networks: initial condensation

TL;DR

The paper analyzes condensation in three-layer neural networks initialized with small weights, revealing a finite-time blow-up in the effective gradient-flow dynamics that contrasts with two-layer behavior. By introducing a final-stage condition, the authors establish a rigorous path to condensation, showing that certain projections diverge and align with a target direction, and they derive an energy upper bound that yields blow-up timing. Complementary experiments corroborate that inner-layer weights converge toward the target direction and that the observed loss plateau corresponds to approaching the final-stage regime. The work further connects condensation to low-rank biases in deep matrix factorization and extends the final-stage concept to matrix completion, highlighting a broader relevance for implicit regularization and network pruning phenomena. Collectively, these results provide a theoretical framework for understanding how small initializations can drive multi-layer networks toward low-complexity, condensed representations with implications for generalization and matrix-factorization tasks.

Abstract

Empirical and theoretical works show that the input weights of two-layer neural networks, when initialized with small values, converge towards isolated orientations. This phenomenon, referred to as condensation, indicates that the gradient descent methods tend to spontaneously reduce the complexity of neural networks during the training process. In this work, we elucidate the mechanisms behind the condensation phenomena occurring in the training of three-layer neural networks and distinguish it from the training of two-layer neural networks. Through rigorous theoretical analysis, we establish the blow-up property of effective dynamics and present a sufficient condition for the occurrence of condensation, findings that are substantiated by experimental results. Additionally, we explore the association between condensation and the low-rank bias observed in deep matrix factorization.

Figures (3)

• Figure 1: Loss curve and condensation phenomenon via gradient descent. The left two plots display the loss changes for the initially trained NN with a learning rate of $5 \times 10^{-3}$ and the retrained NN loaded with the model from iteration $205,000$ using a learning rate of $5 \times 10^{-4}$. The shaded region indicates where effective model fails. The two plots on the right show the cosine similarity of the innermost layer parameters $\bm{W}^{[1]}$ during initialization and condensation respectively.
• Figure 2: Variations of first condition of Assumption \ref{['assump::FinalStageCond']} respect to training process and smaller initialization. The left plot shows the changes of first condition under initialization $\mathcal{N} (0,\frac{1}{200^4})$. The right plot shows the max sum of positive numbers during whole training process under different initialization $\mathcal{N} (0,\frac{1}{200^{2 \alpha}})$which is parametrized by $\alpha$.
• Figure 3: Loss and max singular value. The left and right figures show the changes of loss and the maximum singular value respectively.

Theorems & Definitions (26)

• Proposition 1: conservation laws
• Proof 1
• Remark 1
• Example 1
• Theorem 1: blow up
• Proof 2
• Corollary 1
• Proof 3
• Definition 1: condensed solution
• Proposition 2: induction