Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective

Joel Wendin, Claudio Altafini

TL;DR

The paper analyzes gradient flow dynamics of deep linear networks with quadratic loss by reformulating the training dynamics in terms of a block-shift adjacency matrix $A$, revealing a polynomial, nilpotent, and isospectral ODE with conservation laws. It characterizes a loss landscape with infinitely many global minima and saddle points, none of which are strict local maxima or minima, and introduces a quotient-space viewpoint to classify critical points by invariant loss values. The work further develops simplified dynamics under special initializations (alignment, decoupling, 0-balance, and $\delta$-balanced) and provides detailed analyses of stability, convergence rates, and examples, along with extensions to discrete updates and non-quadratic losses. Collectively, these results offer a global, nonlocal perspective on learning dynamics in deep linear networks, illuminating both the convergence mechanisms and the rich geometry of the loss surface, with potential implications for understanding implicit regularization and initialization effects in broader deep learning contexts.

Abstract

The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size goes to 0) of deep neural networks missing the activation functions and subject to quadratic loss functions. When formulated in terms of the adjacency matrix of the neural network, as we do in the paper, these gradient flow equations form a class of converging matrix ODEs which is nilpotent, polynomial, isospectral, and with conservation laws. The loss landscape is described in detail. It is characterized by infinitely many global minima and saddle points, both strict and nonstrict, but lacks local minima and maxima. The loss function itself is a positive semidefinite Lyapunov function for the gradient flow, and its level sets are unbounded invariant sets of critical points, with critical values that correspond to the amount of singular values of the input-output data learnt by the gradient along a certain trajectory. The adjacency matrix representation we use in the paper allows to highlight the existence of a quotient space structure in which each critical value of the loss function is represented only once, while all other critical points with the same critical value belong to the fiber associated to the quotient space. It also allows to easily determine stable and unstable submanifolds at the saddle points, even when the Hessian fails to obtain them.

Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective

TL;DR

The paper analyzes gradient flow dynamics of deep linear networks with quadratic loss by reformulating the training dynamics in terms of a block-shift adjacency matrix , revealing a polynomial, nilpotent, and isospectral ODE with conservation laws. It characterizes a loss landscape with infinitely many global minima and saddle points, none of which are strict local maxima or minima, and introduces a quotient-space viewpoint to classify critical points by invariant loss values. The work further develops simplified dynamics under special initializations (alignment, decoupling, 0-balance, and -balanced) and provides detailed analyses of stability, convergence rates, and examples, along with extensions to discrete updates and non-quadratic losses. Collectively, these results offer a global, nonlocal perspective on learning dynamics in deep linear networks, illuminating both the convergence mechanisms and the rich geometry of the loss surface, with potential implications for understanding implicit regularization and initialization effects in broader deep learning contexts.

Abstract

The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks, i.e., the gradient descent training dynamics (in the limit when the step size goes to 0) of deep neural networks missing the activation functions and subject to quadratic loss functions. When formulated in terms of the adjacency matrix of the neural network, as we do in the paper, these gradient flow equations form a class of converging matrix ODEs which is nilpotent, polynomial, isospectral, and with conservation laws. The loss landscape is described in detail. It is characterized by infinitely many global minima and saddle points, both strict and nonstrict, but lacks local minima and maxima. The loss function itself is a positive semidefinite Lyapunov function for the gradient flow, and its level sets are unbounded invariant sets of critical points, with critical values that correspond to the amount of singular values of the input-output data learnt by the gradient along a certain trajectory. The adjacency matrix representation we use in the paper allows to highlight the existence of a quotient space structure in which each critical value of the loss function is represented only once, while all other critical points with the same critical value belong to the fiber associated to the quotient space. It also allows to easily determine stable and unstable submanifolds at the saddle points, even when the Hessian fails to obtain them.

Paper Structure

This paper contains 53 sections, 36 theorems, 70 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Problem formulation
  3. Notation
  4. Basic calculus rules for matrices in block-shift form
  5. Gradient flow for deep linear networks: standard formulation
  6. Network formulation of gradient flow
  7. Adjacency matrix dynamics
  8. Conservation laws
  9. Hessian quadratic form
  10. A block-shift singular value decomposition for $A$
  11. Equivariance of gradient flow dynamics to orthogonal transformations
  12. Other properties of $A$
  13. Frobenius norm of $A$: dynamics and equilibria
  14. Adding an $L_2$ regularization
  15. Numerical range of $A$
  16. ...and 38 more sections

Key Result

Proposition 3

Under Assumptions ass:size and ass:distinct-sums, the minimization problem has an infinite number of optimal solutions, and none of them is isolated. If $W_1^\ast ,...,W_h^\ast$ is one such optimal solution, then $\mathcal{L}(W_1^\ast ,W_2^\ast ,...,W_h^\ast)= 0$. Furthermore, the level sets of $\mathcal{L}$ are of dimension $\geq \sum_{i=1}^{h-1} d_i^2$ and unbounded.

Figures (7)

  • Figure 1: Example in Section \ref{['sec:diag-bal']}. (a): Graph $( s_j, \dot{s}_j)$ of the scalar equation \ref{['eq:grad-flow-z']}. Green dots are stable critical points. The origin (in red) is unstable. The lower panel is a zoomed-in version of the upper panel. (b): Singular values of any layer along two gradient flow trajectories. On the upper panel $s_j(0)=s_k(0)$ and the learning is sequential. On the lower panel $s_j(0) \neq s_k (0)$ and the learning is not sequential.
  • Figure 2: Example in Section \ref{['sec:ex-qualitat-anal']}. (a): Case with $A(0)$ near the origin. (b): Case with $A(0)$ away from the origin. Convergence is to a global minimum in both panels.
  • Figure 3: Example of Section \ref{['sec:ex-qualitat-anal']}. Singular values and vectors for $\delta$-balanced trajectories, $\delta = 0.04$. (a): Singular values of $\Sigma$ have relatively close spacing, $\sigma_{j} - \sigma_{j+1} = 1.5$, $\forall j$. The singular vectors approximately align. (b): Singular values of $\Sigma$ are pairwise very close, $\sigma_{j} - \sigma_{j+1} = 0.1$, $j = 1,3$. The singular vectors do not align.
  • Figure 4: Example in Section \ref{['sec:ex-qualitat-anal']}. Arcs $\mathcal{L}(A(\alpha))$ for the various cases of Theorem \ref{['thm:arcs']}.
  • Figure 5: Example of 1 layer with 1 hidden node. Green dots are global minima. The origin in red is a saddle point. (a): Phase space. (b): Loss landscape. Notice the plateau around the origin.
  • ...and 2 more figures

Theorems & Definitions (38)

  • Proposition 3
  • Proposition 4
  • Proposition 5
  • Proposition 6
  • Proposition 7
  • Proposition 8
  • Proposition 9
  • Proposition 10
  • Proposition 11
  • Proposition 12
  • ...and 28 more