Why gradient clipping accelerates training: A theoretical justification for adaptivity

TL;DR

The paper addresses why adaptive gradient methods converge faster in neural network training by introducing a relaxed smoothness condition, (L0,L1)-smoothness, derived from NLP experiments. It proves convergence guarantees for gradient clipping and normalized gradient methods under this condition, including deterministic and stochastic settings, and provides lower bounds showing potential speedups over standard gradient descent. The authors validate the theory with empirical results from language modeling and image classification, demonstrating correlations between local smoothness and gradient norm and illustrating accelerated convergence with clipping. Overall, the work closes part of the gap between practical training heuristics and theoretical guarantees by relaxing the global Lipschitz gradient assumption and analyzing adaptively scaled methods. This offers a theoretical foundation for the observed efficiency of gradient clipping and related techniques in deep learning optimization.

Abstract

We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, \emph{gradient clipping} and \emph{normalized gradient}, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.

Figures (8)

• Figure 1: Gradient norm vs local gradient Lipschitz constant on a log-scale along the training trajectory for AWD-LSTM merity2018regularizing on PTB dataset. The colorbar indicates the number of iterations during training. More experiments can be found in Section \ref{['sec:exp']}. Experiment details are in Appendix \ref{['sec:app-exp']}.
• Figure 2: Gradient norm vs smoothness on log scale for LM training. The dot color indicates the iteration number. Darker ones correspond to earlier iterations. Note that the spans of $x$ and $y$ axis are not fixed.
• Figure 3: Gradient norm vs smoothness on log scale for ResNet20 training. The dot color indicates the iteration number.
• Figure 4: Training and validation loss obtained with different training methods for LSTM and ResNet training. The validation loss plots the cross entropy. The training loss additionally includes the weight regularization term. In the legend, 'lr30clip0.25' denotes that clipped SGD uses step size $30$ and that the $L_2$ norm of the stochastic gradient is clipped by $0.25$. In ResNet training, we threshold the stochastic gradient norm at $0.25$ when clipping is applied.
• Figure 5: Auxiliary plots for Figure \ref{['fig-main_correlation_ptb_withclip']}. The left subfigure shows the values scattered on linear scale. The right subfigure shows more data points from 200 epochs.

Theorems & Definitions (19)

• Definition 1
• Remark 1
• Lemma 2
• proof
• Theorem 3
• Theorem 4
• Remark 5
• Theorem 6
• Theorem 7
• Theorem 8