Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness

Michael Crawshaw, Mingrui Liu

TL;DR

This work addresses the theoretical efficiency of adaptive gradient methods under the relaxed $(L_0,L_1)$-smoothness framework for non-convex stochastic optimization. It develops information-theoretic complexity lower bounds across four adaptive schemes—Decorrelated AdaGrad-Norm, Decorrelated AdaGrad, AdaGrad, and single-step adaptive SGD—showing at least quadratic dependence on problem parameters $(Δ, L_0, L_1)$ and, in many cases, a dominating term that scales with $Δ^2 L_1^2 σ^2/ε^4$, thereby exceeding the optimal $Θ(Δ L σ^2 ε^{-4})$ rate known for $L$-smooth SGD. A key methodological contribution is the construction of difficult objective functions using a piecewise exponential scaffold and high-dimensional adaptations that force divergence or slow progress under relaxed smoothness, plus extensions to affine-noise settings for adaptive SGD. The results imply that, in the relaxed smooth regime, adaptivity can incur fundamental complexity penalties relative to the classical smooth setting, guiding the design and choice of optimization algorithms for non-convex stochastic problems and motivating investigation into algorithms like Adam/AdamW that may circumvent these lower bounds. Overall, the paper clarifies critical limits of adaptation under relaxed smoothness and informs practical algorithm selection for deep learning and related domains.

Abstract

Recent results in non-convex stochastic optimization demonstrate the convergence of popular adaptive algorithms (e.g., AdaGrad) under the $(L_0, L_1)$-smoothness condition, but the rate of convergence is a higher-order polynomial in terms of problem parameters like the smoothness constants. The complexity guaranteed by such algorithms to find an $ε$-stationary point may be significantly larger than the optimal complexity of $Θ\left( ΔL σ^2 ε^{-4} \right)$ achieved by SGD in the $L$-smooth setting, where $Δ$ is the initial optimality gap, $σ^2$ is the variance of stochastic gradient. However, it is currently not known whether these higher-order dependencies can be tightened. To answer this question, we investigate complexity lower bounds for several adaptive optimization algorithms in the $(L_0, L_1)$-smooth setting, with a focus on the dependence in terms of problem parameters $Δ, L_0, L_1$. We provide complexity bounds for three variations of AdaGrad, which show at least a quadratic dependence on problem parameters $Δ, L_0, L_1$. Notably, we show that the decorrelated variant of AdaGrad-Norm requires at least $Ω\left( Δ^2 L_1^2 σ^2 ε^{-4} \right)$ stochastic gradient queries to find an $ε$-stationary point. We also provide a lower bound for SGD with a broad class of adaptive stepsizes. Our results show that, for certain adaptive algorithms, the $(L_0, L_1)$-smooth setting is fundamentally more difficult than the standard smooth setting, in terms of the initial optimality gap and the smoothness constants.

Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness

TL;DR

This work addresses the theoretical efficiency of adaptive gradient methods under the relaxed -smoothness framework for non-convex stochastic optimization. It develops information-theoretic complexity lower bounds across four adaptive schemes—Decorrelated AdaGrad-Norm, Decorrelated AdaGrad, AdaGrad, and single-step adaptive SGD—showing at least quadratic dependence on problem parameters and, in many cases, a dominating term that scales with , thereby exceeding the optimal rate known for -smooth SGD. A key methodological contribution is the construction of difficult objective functions using a piecewise exponential scaffold and high-dimensional adaptations that force divergence or slow progress under relaxed smoothness, plus extensions to affine-noise settings for adaptive SGD. The results imply that, in the relaxed smooth regime, adaptivity can incur fundamental complexity penalties relative to the classical smooth setting, guiding the design and choice of optimization algorithms for non-convex stochastic problems and motivating investigation into algorithms like Adam/AdamW that may circumvent these lower bounds. Overall, the paper clarifies critical limits of adaptation under relaxed smoothness and informs practical algorithm selection for deep learning and related domains.

Abstract

Recent results in non-convex stochastic optimization demonstrate the convergence of popular adaptive algorithms (e.g., AdaGrad) under the -smoothness condition, but the rate of convergence is a higher-order polynomial in terms of problem parameters like the smoothness constants. The complexity guaranteed by such algorithms to find an -stationary point may be significantly larger than the optimal complexity of achieved by SGD in the -smooth setting, where is the initial optimality gap, is the variance of stochastic gradient. However, it is currently not known whether these higher-order dependencies can be tightened. To answer this question, we investigate complexity lower bounds for several adaptive optimization algorithms in the -smooth setting, with a focus on the dependence in terms of problem parameters . We provide complexity bounds for three variations of AdaGrad, which show at least a quadratic dependence on problem parameters . Notably, we show that the decorrelated variant of AdaGrad-Norm requires at least stochastic gradient queries to find an -stationary point. We also provide a lower bound for SGD with a broad class of adaptive stepsizes. Our results show that, for certain adaptive algorithms, the -smooth setting is fundamentally more difficult than the standard smooth setting, in terms of the initial optimality gap and the smoothness constants.
Paper Structure (29 sections, 24 theorems, 280 equations, 1 figure, 1 table)

This paper contains 29 sections, 24 theorems, 280 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Optimization Objectives
  5. Optimization Algorithms
  6. Complexity
  7. Decorrelated AdaGrad-Norm
  8. Proof Outline
  9. AdaGrad and Decorrelated AdaGrad
  10. Proof Outline
  11. Single-Step Adaptive SGD
  12. Proof Outline
  13. Discussion and Conclusion
  14. Proof of Theorem \ref{['thm:adagrad_norm']}
  15. Case 1
  16. ...and 14 more sections

Key Result

Theorem 1

Denote ${\mathcal{F}} = {\mathcal{F}}_{\textup{as}}(\Delta, L_0, L_1, \sigma)$, and let algorithm $A_{\text{DAN}}$ denote Decorrelated AdaGrad-Norm (Equation eq:adagrad_norm) with parameters $\eta > 0$ and $0 < \gamma \leq \tilde{\mathcal{O}} \left( \Delta L_1 \right)$. Let $0 < \epsilon \leq \mathc

Figures (1)

  • Figure 1: Objectives from Lemma \ref{['lem:adagrad_norm_div']}. $m := (\psi')^{-1}(\Delta L_1) = \frac{1}{L_1} \log \left( 1 + \frac{\Delta L_1^2}{L_0} \right)$.

Theorems & Definitions (41)

  • Theorem 1
  • Lemma 1
  • Lemma 2
  • Theorem 2
  • Theorem 3
  • Lemma 3
  • Lemma 4
  • Theorem 4
  • Lemma 5: Restatement of Lemma \ref{['lem:adagrad_norm_div']}
  • proof
  • ...and 31 more