Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

Xiaoyu Li, Francesco Orabona

TL;DR

The paper analyzes generalized AdaGrad-style adaptive stepsizes for stochastic gradient descent in both convex and non-convex settings. It proves almost-sure convergence of gradient norms to zero for non-convex objectives and establishes adaptive finite-time convergence rates in the convex and non-convex cases, showing the step sizes automatically adjust to the noise level and interpolate between $O(1/T)$ (gradient descent) and $O(1/\sqrt{T})$ (SGD) up to polylog terms. A key theoretical contribution is the first non-convex guarantee for generalized AdaGrad stepsizes, along with a demonstration that keeping the update direction unbiased is essential. These results provide a principled explanation for when and why AdaGrad-like stepsizes can outperform plain SGD in non-convex optimization, without requiring prior knowledge of the gradient noise variance. The work also avoids strong bounded-domain projections and moves closer to aligning theory with practical, unconstrained optimization in machine learning.

Abstract

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in our theoretical understanding of these methods, especially in the non-convex setting. In this paper, we start closing this gap: we theoretically analyze in the convex and non-convex settings a generalized version of the AdaGrad stepsizes. We show sufficient conditions for these stepsizes to achieve almost sure asymptotic convergence of the gradients to zero, proving the first guarantee for generalized AdaGrad stepsizes in the non-convex setting. Moreover, we show that these stepsizes allow to automatically adapt to the level of noise of the stochastic gradients in both the convex and non-convex settings, interpolating between $O(1/T)$ and $O(1/\sqrt{T})$, up to logarithmic terms.

On the Convergence of Stochastic Gradient Descent with Adaptive Stepsizes

TL;DR

The paper analyzes generalized AdaGrad-style adaptive stepsizes for stochastic gradient descent in both convex and non-convex settings. It proves almost-sure convergence of gradient norms to zero for non-convex objectives and establishes adaptive finite-time convergence rates in the convex and non-convex cases, showing the step sizes automatically adjust to the noise level and interpolate between (gradient descent) and (SGD) up to polylog terms. A key theoretical contribution is the first non-convex guarantee for generalized AdaGrad stepsizes, along with a demonstration that keeping the update direction unbiased is essential. These results provide a principled explanation for when and why AdaGrad-like stepsizes can outperform plain SGD in non-convex optimization, without requiring prior knowledge of the gradient noise variance. The work also avoids strong bounded-domain projections and moves closer to aligning theory with practical, unconstrained optimization in machine learning.

Abstract

Stochastic gradient descent is the method of choice for large scale optimization of machine learning objective functions. Yet, its performance is greatly variable and heavily depends on the choice of the stepsizes. This has motivated a large body of research on adaptive stepsizes. However, there is currently a gap in our theoretical understanding of these methods, especially in the non-convex setting. In this paper, we start closing this gap: we theoretically analyze in the convex and non-convex settings a generalized version of the AdaGrad stepsizes. We show sufficient conditions for these stepsizes to achieve almost sure asymptotic convergence of the gradients to zero, proving the first guarantee for generalized AdaGrad stepsizes in the non-convex setting. Moreover, we show that these stepsizes allow to automatically adapt to the level of noise of the stochastic gradients in both the convex and non-convex settings, interpolating between and , up to logarithmic terms.

Paper Structure

This paper contains 18 sections, 14 theorems, 70 equations.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. PROBLEM SET-UP
  4. Notation.
  5. Setting and Assumptions.
  6. KEEPING THE UPDATE DIRECTION UNBIASED
  7. ALMOST SURE CONVERGENCE FOR NON-CONVEX FUNCTIONS
  8. ADAPTIVE CONVERGENCE RATES
  9. Adaptive Convergence for Convex Functions
  10. Remark.
  11. Adaptive Convergence for Non-Convex Functions
  12. Remark.
  13. DISCUSSION AND FUTURE WORK
  14. Appendix
  15. Details of Example \ref{['ex:ninety']}
  16. ...and 3 more sections

Key Result

Theorem 1

Assume (H1, H2, H3, H4). The stepsizes are chosen as in eq:eta, where $\alpha,\beta>0$ and $\epsilon \in (0,\frac{1}{2}]$. Then, the gradients of SGD converges to zero almost surely. Moreover, $\lim\inf_{t\rightarrow \infty} \|\nabla f(\boldsymbol{x}_t)\|^2 t^{1/2-\epsilon}=0$ almost surely.

Theorems & Definitions (29)

  • Example 1
  • Theorem 1
  • Theorem 2
  • Lemma 1
  • Lemma 2
  • Lemma 3
  • proof : Proof of Theorem \ref{['thm:convergence_sgd']}
  • Theorem 3
  • Lemma 4
  • Lemma 5
  • ...and 19 more