Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness

Thomas Pethick, Wanyun Xie, Mete Erdogan, Kimon Antonakopoulos, Tony Silveti-Falls, Volkan Cevher

TL;DR

The paper addresses the instability of first-order methods in non-Euclidean spaces by blending gradient-norm clipping with conditional gradient steps, extending clipping to general (L0,L1)-smooth settings. It introduces Generalized Gradient Norm Clipping (GGNC) and stochastic variants (uSCG, S^3CG) with momentum, proving descent properties and an order-optimal O(n^{-1/4}) convergence rate in stochastic regimes. By connecting clipping to the Frank-Wolfe short-step, the work enables principled incorporation of weight decay and constrained updates, with theoretical guarantees. The methods are instantiated for deep learning (ClippedScion) and validated on image classification and language modeling tasks, showing practical speedups and stability benefits across norm choices and network architectures.

Abstract

This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of ($L_0$,$L_1$)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal $O(n^{-1/4})$ convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

Generalized Gradient Norm Clipping & Non-Euclidean $(L_0,L_1)$-Smoothness

TL;DR

The paper addresses the instability of first-order methods in non-Euclidean spaces by blending gradient-norm clipping with conditional gradient steps, extending clipping to general (L0,L1)-smooth settings. It introduces Generalized Gradient Norm Clipping (GGNC) and stochastic variants (uSCG, S^3CG) with momentum, proving descent properties and an order-optimal O(n^{-1/4}) convergence rate in stochastic regimes. By connecting clipping to the Frank-Wolfe short-step, the work enables principled incorporation of weight decay and constrained updates, with theoretical guarantees. The methods are instantiated for deep learning (ClippedScion) and validated on image classification and language modeling tasks, showing practical speedups and stability benefits across norm choices and network architectures.

Abstract

This work introduces a hybrid non-Euclidean optimization method which generalizes gradient norm clipping by combining steepest descent and conditional gradient approaches. The method achieves the best of both worlds by establishing a descent property under a generalized notion of (,)-smoothness. Weight decay is incorporated in a principled manner by identifying a connection to the Frank-Wolfe short step. In the stochastic case, we show an order optimal convergence rate by leveraging a momentum based gradient estimator. We discuss how to instantiate the algorithms for deep learning, which we dub Clipped Scion, and demonstrate their properties on image classification and language modeling. The code is available at https://github.com/LIONS-EPFL/ClippedScion.

Paper Structure

This paper contains 44 sections, 4 theorems, 138 equations, 11 figures, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Preliminaries
  4. Observation I
  5. Observation II
  6. Method
  7. Stochastic case
  8. Weight decay & constrained problems
  9. Norm choices
  10. Sign
  11. Spectral
  12. Product norm
  13. Analysis
  14. Deterministic case
  15. Deterministic case
  16. ...and 29 more sections

Key Result

lemma 1

Suppose asm:Lipasm:stoch hold and let $n\in\mathbb{N}^*$. Consider the iterates $\{x^k\}_{1\leq k \leq n}$ generated by alg:GGNC. Then, for all $k\in\{1,\ldots,n \}$, it holds where $\zeta_* := \max_{x\in\mathcal{X}}\frac{\left\Vert{x}\right\Vert_2}{\left\Vert{x}\right\Vert_{\ast}}$.

Figures (11)

  • Figure 1: The optimal hyperparameters for Unconstrained ClippedScion on CIFAR10 for 80 epochs, (left) no stepsize decay (right) with stepsize decay. (indicated in red). The first row indicated with "No Clip" corresponds to Unconstrained Scion.
  • Figure 2: The optimal hyperparameters for ClippedScion on CIFAR10 for 80 epochs, no stepsize decay (indicated in red). The first row indicated with "No Clip" corresponds to Scion.
  • Figure 3: For CIFAR10 experiments with stepsize decay; Unconstrained Scion and Unconstrained ClippedScion achieve similar performance as expected.
  • Figure 4: For CIFAR10 experiments for constrained variant of the algorithms without stepsize decay; clipping is less effective due to the surprising increase of $\braket{v^k,d^k}$. We observe that even the (deterministic) Wolfe gap is increasing, which is otherwise expected to go to zero.
  • Figure 5: Clipping improves over Scion by a 11% speedup on DeiT-base.
  • ...and 6 more figures

Theorems & Definitions (12)

  • Definition 1
  • Remark 3
  • Remark 4
  • lemma 1: Linear recursive inequality for $\mathbb{E}\left\Vert{\lambda^k}\right\Vert_2^2$ for GGNC
  • proof
  • lemma 2: Bound on $\mathbb{E}\|\lambda^k\|_2^2$ with horizon-dependent $\alpha$ for GGNC
  • proof
  • Remark 6
  • lemma 3: Linear recursive inequality for $\mathbb{E}\left\Vert{\lambda^k}\right\Vert_2^2$ for S$^3$CG
  • proof
  • ...and 2 more