Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Random Scaling and Momentum for Non-smooth Non-convex Optimization

Qinzi Zhang, Ashok Cutkosky

TL;DR

This work addresses optimization of non-smooth non-convex objectives typical in neural network training by introducing a relaxed convergence criterion, the $(c,\epsilon)$-stationary point, and a general Exponentiated Online-to-non-convex (O2NC) framework. A key idea is to random-scale online updates with an exponential factor, enabling a direct online-to-non-convex reduction that connects online convex optimization to non-convex stochastic optimization. Instantiating the framework with a simple unconstrained online gradient descent yields a SGDM-like algorithm with an exponential random scaling on the update, and the authors prove optimal convergence rates under their criterion, including matching lower bounds. Empirical results on CIFAR-10 with ResNet-18 show the randomized-SGDM variant performs comparably to standard SGDM, supporting the theoretical claims and suggesting practical viability. The work further highlights the potential to derive adaptive variants (e.g., Adam-like behavior) within this framework, opening avenues for future research in adaptive non-convex optimization.

Abstract

Training neural networks requires optimizing a loss function that may be highly irregular, and in particular neither convex nor smooth. Popular training algorithms are based on stochastic gradient descent with momentum (SGDM), for which classical analysis applies only if the loss is either convex or smooth. We show that a very small modification to SGDM closes this gap: simply scale the update at each time point by an exponentially distributed random scalar. The resulting algorithm achieves optimal convergence guarantees. Intriguingly, this result is not derived by a specific analysis of SGDM: instead, it falls naturally out of a more general framework for converting online convex optimization algorithms to non-convex optimization algorithms.

Random Scaling and Momentum for Non-smooth Non-convex Optimization

TL;DR

This work addresses optimization of non-smooth non-convex objectives typical in neural network training by introducing a relaxed convergence criterion, the -stationary point, and a general Exponentiated Online-to-non-convex (O2NC) framework. A key idea is to random-scale online updates with an exponential factor, enabling a direct online-to-non-convex reduction that connects online convex optimization to non-convex stochastic optimization. Instantiating the framework with a simple unconstrained online gradient descent yields a SGDM-like algorithm with an exponential random scaling on the update, and the authors prove optimal convergence rates under their criterion, including matching lower bounds. Empirical results on CIFAR-10 with ResNet-18 show the randomized-SGDM variant performs comparably to standard SGDM, supporting the theoretical claims and suggesting practical viability. The work further highlights the potential to derive adaptive variants (e.g., Adam-like behavior) within this framework, opening avenues for future research in adaptive non-convex optimization.

Abstract

Training neural networks requires optimizing a loss function that may be highly irregular, and in particular neither convex nor smooth. Popular training algorithms are based on stochastic gradient descent with momentum (SGDM), for which classical analysis applies only if the loss is either convex or smooth. We show that a very small modification to SGDM closes this gap: simply scale the update at each time point by an exponentially distributed random scalar. The resulting algorithm achieves optimal convergence guarantees. Intriguingly, this result is not derived by a specific analysis of SGDM: instead, it falls naturally out of a more general framework for converting online convex optimization algorithms to non-convex optimization algorithms.
Paper Structure (33 sections, 18 theorems, 85 equations, 1 figure, 1 table, 2 algorithms)

This paper contains 33 sections, 18 theorems, 85 equations, 1 figure, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. Preliminaries
  4. Notations
  5. Stochastic Optimization
  6. Online Learning
  7. Non-smooth Optimization
  8. Online-to-non-convex Conversion
  9. Paper Organization
  10. Exponentiated Online-to-non-convex
  11. Random Scaling
  12. Exponentiated and Regularized Losses
  13. Exponentially Weighted Gradients
  14. Variance Regularization
  15. Convergence Analysis
  16. ...and 18 more sections

Key Result

Lemma 2.2

Suppose $F$ is $H$-smooth. If $\|\nabla F({\boldsymbol x})\|_c\le\epsilon$ where $c=H^2\epsilon^{-1}$, then $\|\nabla F({\boldsymbol x})\|\le 2\epsilon$. Suppose $F$ is $\rho$-second-order-smooth. If $\|\nabla F({\boldsymbol x})\|_c\le\epsilon$ where $c=\rho/2$, then $\|\nabla F({\boldsymbol x})\|\l

Figures (1)

  • Figure 1: Experiments on CIFAR-10 with ResNet-18 Network. The curves represent the average performance of each optimizer in three trials, and the shaded regions denote the standard deviation.

Theorems & Definitions (29)

  • Definition 2.2
  • Lemma 2.2
  • Lemma 2.2
  • Lemma 3.0
  • Lemma 3.0
  • Theorem 3.1
  • Theorem 4.1
  • Theorem 4.2
  • Corollary 5.1
  • Lemma 1.0
  • ...and 19 more