Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence of gradient descent for deep neural networks

Sourav Chatterjee

TL;DR

A probabilistic corollary for random initializations is discussed, its dependence on the probability of the required initialization event is clarified, and numerical experiments showing that this theory-guided initialization can substantially accelerate optimization relative to standard random initializations at the same width are provided.

Abstract

We give a simple local Polyak-Lojasiewicz (PL) criterion that guarantees linear (exponential) convergence of gradient flow and gradient descent to a zero-loss solution of a nonnegative objective. We then verify this criterion for the squared training loss of a feedforward neural network with smooth, strictly increasing activation functions, in a regime that is complementary to the usual over-parameterized analyses: the network width and depth are fixed, while the input data vectors are assumed to be linearly independent (in particular, the ambient input dimension is at least the number of data points). A notable feature of the verification is that it is constructive: it leads to a simple "positive" initialization (zero first-layer weights, strictly positive hidden-layer weights, and sufficiently large output-layer weights) under which gradient descent provably converges to an interpolating global minimizer of the training loss. We also discuss a probabilistic corollary for random initializations, clarify its dependence on the probability of the required initialization event, and provide numerical experiments showing that this theory-guided initialization can substantially accelerate optimization relative to standard random initializations at the same width.

Convergence of gradient descent for deep neural networks

TL;DR

A probabilistic corollary for random initializations is discussed, its dependence on the probability of the required initialization event is clarified, and numerical experiments showing that this theory-guided initialization can substantially accelerate optimization relative to standard random initializations at the same width are provided.

Abstract

We give a simple local Polyak-Lojasiewicz (PL) criterion that guarantees linear (exponential) convergence of gradient flow and gradient descent to a zero-loss solution of a nonnegative objective. We then verify this criterion for the squared training loss of a feedforward neural network with smooth, strictly increasing activation functions, in a regime that is complementary to the usual over-parameterized analyses: the network width and depth are fixed, while the input data vectors are assumed to be linearly independent (in particular, the ambient input dimension is at least the number of data points). A notable feature of the verification is that it is constructive: it leads to a simple "positive" initialization (zero first-layer weights, strictly positive hidden-layer weights, and sufficiently large output-layer weights) under which gradient descent provably converges to an interpolating global minimizer of the training loss. We also discuss a probabilistic corollary for random initializations, clarify its dependence on the probability of the required initialization event, and provide numerical experiments showing that this theory-guided initialization can substantially accelerate optimization relative to standard random initializations at the same width.
Paper Structure (18 sections, 17 theorems, 106 equations, 3 figures)

This paper contains 18 sections, 17 theorems, 106 equations, 3 figures.

Table of Contents

  1. A convergence criterion for gradient descent
  2. Background
  3. Main results
  4. How the criterion will be used for neural networks
  5. Application to deep neural networks
  6. Feedforward neural networks
  7. A high-dimensional data regime
  8. Our results for feedforward neural networks
  9. Numerical experiments
  10. Setup.
  11. Initializations compared.
  12. Baseline result (width $d_1=d_2=5$).
  13. Wider network.
  14. Output-layer scaling.
  15. Proof of Theorem \ref{['mainthm2']}
  16. ...and 3 more sections

Key Result

Theorem 1.1

Let $f$, $x_0$, and $\alpha$ be as above. Assume that mainassump holds for some $r>0$, and let $\alpha := \alpha(x_0,r)$. Then there is a unique solution of the gradient flow equation on $[0,\infty)$ with $\phi(0)=x_0$, and this flow stays in $B(x_0,r)$ for all time, and converges to some $x^*\in B(x_0,r)$ where $f(x^*)=0$. Moreover, for each $t\ge 0$, we have

Figures (3)

  • Figure 1: Mean training loss (MSE) versus gradient descent iteration in the baseline experiment. The y-axis is logarithmic. The legend is placed outside the plotting region and the curves use distinct line types so that color is not essential.
  • Figure 2: Mean training loss (MSE) versus gradient descent iteration for the same synthetic setup, but with widths $d_1=d_2=10$. The y-axis is logarithmic.
  • Figure 3: Mean training loss (MSE) versus iteration for the positive initialization with different output-layer scales $A$. The y-axis is logarithmic.

Theorems & Definitions (40)

  • Theorem 1.1
  • Theorem 1.2
  • Remark 1.3: Beyond $C^2$ smoothness
  • Theorem 2.1
  • Remark 2.2: Width versus ambient dimension
  • Remark 2.3: Why the condition $W_1=0$ is not vacuous
  • Remark 2.4: Why require hidden-layer weights to be positive?
  • Remark 2.5: Activation shifts, biases, and smooth leaky-ReLU
  • Remark 2.6: Explicit constants in Theorem \ref{['deepthm']}
  • Remark 2.7: Relation to "lazy training" results
  • ...and 30 more