Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum

Difei Cheng, Ruinan Jin, Hong Qiao, Bo Zhang

TL;DR

The paper addresses the problem of last-iterate convergence for distributed momentum SGD in non-convex settings under Robbins–Monro step sizes $\epsilon_n$. It develops a unified momentum-based framework for PSASGD, EASGD, and D-PSGD, and proves both almost-sure and mean-square convergence of the last iterate to stationary points, with explicit convergence rates. A key finding is that momentum does not improve the asymptotic rate when $\epsilon_n=\frac{\sqrt{m}}{\sqrt{n}}$, but larger momentum $\alpha$ accelerates early-stage convergence toward a neighborhood of stationary points, as quantified by probabilistic bounds. Experiments on CIFAR-10/100 with ResNet-20 corroborate the theory, showing faster reduction of the gradient norm for higher momentum and validating the practical benefits for privacy-preserving distributed training.

Abstract

Distributed stochastic gradient methods are widely used to preserve data privacy and ensure scalability in large-scale learning tasks. While existing theory on distributed momentum Stochastic Gradient Descent (mSGD) mainly focuses on time-averaged convergence, the more practical last-iterate convergence remains underexplored. In this work, we analyze the last-iterate convergence behavior of distributed mSGD in non-convex settings under the classical Robbins-Monro step-size schedule. We prove both almost sure convergence and $L_2$ convergence of the last iterate, and derive convergence rates. We further show that momentum can accelerate early-stage convergence, and provide experiments to support our theory.

Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum

TL;DR

The paper addresses the problem of last-iterate convergence for distributed momentum SGD in non-convex settings under Robbins–Monro step sizes . It develops a unified momentum-based framework for PSASGD, EASGD, and D-PSGD, and proves both almost-sure and mean-square convergence of the last iterate to stationary points, with explicit convergence rates. A key finding is that momentum does not improve the asymptotic rate when , but larger momentum accelerates early-stage convergence toward a neighborhood of stationary points, as quantified by probabilistic bounds. Experiments on CIFAR-10/100 with ResNet-20 corroborate the theory, showing faster reduction of the gradient norm for higher momentum and validating the practical benefits for privacy-preserving distributed training.

Abstract

Distributed stochastic gradient methods are widely used to preserve data privacy and ensure scalability in large-scale learning tasks. While existing theory on distributed momentum Stochastic Gradient Descent (mSGD) mainly focuses on time-averaged convergence, the more practical last-iterate convergence remains underexplored. In this work, we analyze the last-iterate convergence behavior of distributed mSGD in non-convex settings under the classical Robbins-Monro step-size schedule. We prove both almost sure convergence and convergence of the last iterate, and derive convergence rates. We further show that momentum can accelerate early-stage convergence, and provide experiments to support our theory.
Paper Structure (17 sections, 9 theorems, 84 equations, 1 figure)

This paper contains 17 sections, 9 theorems, 84 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Main Results
  3. Definitions of Convergence
  4. General Momentum-Based Iteration
  5. D-PSGD.
  6. EASGD.
  7. Momentum Extension.
  8. Last-iterate Convergence
  9. Last-iterate convergence rate
  10. Experiments Results
  11. Conclusion
  12. Biography Section
  13. Appendix
  14. Useful lemmas
  15. Proof of Theorem \ref{['lem_mied']}
  16. ...and 2 more sections

Key Result

Theorem 2.1

Suppose $\{X_{n}\}$ is a sequence generated by equation 123combinemomentum. Under Assumptions ass_g1--opjknhg, it holds that $\|\nabla g(\overline{x}_{n})\|\rightarrow 0\ a.s.$ and $\mathop{\mathrm{\mathbb{E}}}\nolimits\|\nabla g(\overline{x}_{n})\|^{2}\rightarrow 0,$ where $\overline{x}_{n}$ is d

Figures (1)

  • Figure 1: Training and prediction performance on CIFAR-10 and CIFAR-100 with 1,3,10,20 sub-datasets (workers). (a)-(d): The training loss with 1, 3, 10, and 20 sub-datasets respectively on CIFAR-10. (e)-(h): The accuracy with 1, 3, 10, and 20 sub-datasets respectively on CIFAR-10. (i)-(l): The training loss with 1, 3, 10, and 20 sub-datasets respectively on CIFAR-100. (m)-(p): The accuracy with 1, 3, 10, and 20 sub-datasets respectively on CIFAR-100.

Theorems & Definitions (11)

  • Theorem 2.1
  • Theorem 2.2
  • Theorem 2.3
  • Lemma A.1
  • Lemma A.2
  • Lemma A.3
  • Lemma A.4
  • Lemma A.5
  • Lemma A.6
  • proof
  • ...and 1 more