Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization

Kwangjun Ahn, Gagik Magakyan, Ashok Cutkosky

TL;DR

It is shown that schedule-free SGD achieves optimal iteration complexity for nonsmooth, nonconvex optimization problems, addressing a theoretical gap that the convex theory cannot explain.

Abstract

This work investigates the effectiveness of schedule-free methods, developed by A. Defazio et al. (NeurIPS 2024), in nonconvex optimization settings, inspired by their remarkable empirical success in training neural networks. Specifically, we show that schedule-free SGD achieves optimal iteration complexity for nonsmooth, nonconvex optimization problems. Our proof begins with the development of a general framework for online-to-nonconvex conversion, which converts a given online learning algorithm into an optimization algorithm for nonconvex losses. Our general framework not only recovers existing conversions but also leads to two novel conversion schemes. Notably, one of these new conversions corresponds directly to schedule-free SGD, allowing us to establish its optimality. Additionally, our analysis provides valuable insights into the parameter choices for schedule-free SGD, addressing a theoretical gap that the convex theory cannot explain.

General framework for online-to-nonconvex conversion: Schedule-free SGD is also effective for nonconvex optimization

TL;DR

It is shown that schedule-free SGD achieves optimal iteration complexity for nonsmooth, nonconvex optimization problems, addressing a theoretical gap that the convex theory cannot explain.

Abstract

This work investigates the effectiveness of schedule-free methods, developed by A. Defazio et al. (NeurIPS 2024), in nonconvex optimization settings, inspired by their remarkable empirical success in training neural networks. Specifically, we show that schedule-free SGD achieves optimal iteration complexity for nonsmooth, nonconvex optimization problems. Our proof begins with the development of a general framework for online-to-nonconvex conversion, which converts a given online learning algorithm into an optimization algorithm for nonconvex losses. Our general framework not only recovers existing conversions but also leads to two novel conversion schemes. Notably, one of these new conversions corresponds directly to schedule-free SGD, allowing us to establish its optimality. Additionally, our analysis provides valuable insights into the parameter choices for schedule-free SGD, addressing a theoretical gap that the convex theory cannot explain.

Paper Structure

This paper contains 37 sections, 18 theorems, 109 equations, 1 figure, 7 algorithms.

Table of Contents

  1. Introduction
  2. Our main result and approach
  3. Related work
  4. Preliminaries
  5. Assumptions on the loss function and stochastic gradients
  6. Approximate Goldstein stationary point
  7. Online learning
  8. General framework for online-to-nonconvex conversion
  9. Output of the general scheme
  10. Conversion guarantees
  11. User-friendly nonconvex optimization guarantees
  12. Warm-up: two simple ways to achieve optimal nonconvex guarantees
  13. Option I leads to previous conversion
  14. Option II leads to a novel optimal conversion
  15. Schedule-free SGD is effective for nonconvex optimization
  16. ...and 22 more sections

Key Result

Theorem 1

Schedule-free SGD schefree, with an appropriate choice of parameters $\gamma, c_t, \kappa_t$, achieves optimal rates for nonsmooth and nonconvex $F$.

Figures (1)

  • Figure 1: Illustration of how \ref{['alg:SF']} can be interpreted as schedule-free SGD. By defining the ${\mathbf z}$-iterates according to \ref{['def:zt']}, it becomes clear that the ${\mathbf z}$-iterates follow the base SGD trajectory of schedule-free SGD \ref{['schefree']}.

Theorems & Definitions (36)

  • Theorem 1: Informal; see \ref{['sec:SF']}
  • Definition 1: $(\lambda,\varepsilon)$-stationary point
  • Proposition 1
  • Definition 2: Discounted regret
  • Definition 3: Exponential moving average (EMA) of iterates
  • Definition 4: Random index distribution
  • Definition 5: ${\mathbf x}$-iterate stability
  • Lemma 1: Generic online-to-nonconvex conversion
  • proof
  • Lemma 2
  • ...and 26 more