Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revisiting Convergence of AdaGrad with Relaxed Assumptions

Yusu Hong, Junhong Lin

TL;DR

The paper analyzes AdaGrad with momentum for non-convex stochastic optimization under a relaxed noise model that covers affine-variance and expected-smoothness scenarios. It introduces a proxy-step-size technique to decouple adaptive scaling from stochastic gradients and derives high-probability convergence bounds that are adaptive to noise parameters, achieving $\tilde{O}(1/\sqrt{T})$ and, under small noise, $\tilde{O}(1/\!T)$ rates without requiring prior problem-parameter knowledge. The framework extends to a generalized $(L_0,L_1)$-smoothness setting, preserving the same order of convergence while highlighting the need for parameter tuning in unbounded-smooth contexts. The results bridge gaps between lower bounds for stochastic first-order methods and adaptive-coordinate methods, offering practical implications for training with sparse or noisy gradients. Limitations point to extending the analysis to other adaptive optimizers and providing additional empirical validation under the relaxed assumptions.

Abstract

In this study, we revisit the convergence of AdaGrad with momentum (covering AdaGrad as a special case) on non-convex smooth optimization problems. We consider a general noise model where the noise magnitude is controlled by the function value gap together with the gradient magnitude. This model encompasses a broad range of noises including bounded noise, sub-Gaussian noise, affine variance noise and the expected smoothness, and it has been shown to be more realistic in many practical applications. Our analysis yields a probabilistic convergence rate which, under the general noise, could reach at (\tilde{\mathcal{O}}(1/\sqrt{T})). This rate does not rely on prior knowledge of problem-parameters and could accelerate to (\tilde{\mathcal{O}}(1/T)) where (T) denotes the total number iterations, when the noise parameters related to the function value gap and noise level are sufficiently small. The convergence rate thus matches the lower rate for stochastic first-order methods over non-convex smooth landscape up to logarithm terms [Arjevani et al., 2023]. We further derive a convergence bound for AdaGrad with mometum, considering the generalized smoothness where the local smoothness is controlled by a first-order function of the gradient norm.

Revisiting Convergence of AdaGrad with Relaxed Assumptions

TL;DR

The paper analyzes AdaGrad with momentum for non-convex stochastic optimization under a relaxed noise model that covers affine-variance and expected-smoothness scenarios. It introduces a proxy-step-size technique to decouple adaptive scaling from stochastic gradients and derives high-probability convergence bounds that are adaptive to noise parameters, achieving and, under small noise, rates without requiring prior problem-parameter knowledge. The framework extends to a generalized -smoothness setting, preserving the same order of convergence while highlighting the need for parameter tuning in unbounded-smooth contexts. The results bridge gaps between lower bounds for stochastic first-order methods and adaptive-coordinate methods, offering practical implications for training with sparse or noisy gradients. Limitations point to extending the analysis to other adaptive optimizers and providing additional empirical validation under the relaxed assumptions.

Abstract

In this study, we revisit the convergence of AdaGrad with momentum (covering AdaGrad as a special case) on non-convex smooth optimization problems. We consider a general noise model where the noise magnitude is controlled by the function value gap together with the gradient magnitude. This model encompasses a broad range of noises including bounded noise, sub-Gaussian noise, affine variance noise and the expected smoothness, and it has been shown to be more realistic in many practical applications. Our analysis yields a probabilistic convergence rate which, under the general noise, could reach at (\tilde{\mathcal{O}}(1/\sqrt{T})). This rate does not rely on prior knowledge of problem-parameters and could accelerate to (\tilde{\mathcal{O}}(1/T)) where (T) denotes the total number iterations, when the noise parameters related to the function value gap and noise level are sufficiently small. The convergence rate thus matches the lower rate for stochastic first-order methods over non-convex smooth landscape up to logarithm terms [Arjevani et al., 2023]. We further derive a convergence bound for AdaGrad with mometum, considering the generalized smoothness where the local smoothness is controlled by a first-order function of the gradient norm.
Paper Structure (41 sections, 25 theorems, 142 equations, 1 figure, 1 table, 1 algorithm)

This paper contains 41 sections, 25 theorems, 142 equations, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contribution
  3. Related works
  4. Convergence of AdaGrad
  5. Convergence with affine variance noise
  6. Convergence with the expected smoothness
  7. Convergence with generalized smoothness
  8. Problem setting and algorithm
  9. Notations
  10. Assumption
  11. AdaGrad with momentum
  12. AdaGrad
  13. Main Convergence Result
  14. Convergence of AdaGrad with affine variance noise
  15. Proof detail
  16. ...and 26 more sections

Key Result

Theorem 1

Given $T \ge 1$, let $\{\bm{x}_s\}_{s \in [T]}$ be generated by Algorithm alg:AdaGrad. If Assumptions (A1), (A2), (A3) hold, then for any $\beta \in [0,1),\eta,\epsilon > 0$ and $\delta \in (0,1)$, it holds that with probability at least $1-\delta$, where $B_1=B+1,$${\Delta}_1 = {\Delta}(1-\beta)/\eta$, and ${\Delta}$ is given byThe detailed expression of ${\Delta}$ could be found in eq:define_tD

Figures (1)

  • Figure 1: Training loss vs. steps and gradient norms vs. steps using SGD and AdaGrad.

Theorems & Definitions (45)

  • Theorem 1
  • Remark 4.1
  • Corollary 1
  • Remark 4.2
  • Lemma 5.1
  • Lemma 5.2
  • Lemma 5.3
  • Lemma 5.4
  • Lemma 5.5
  • Lemma 5.6
  • ...and 35 more