Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization

Fan Zhou, Guojing Cong

TL;DR

The paper analyzes a synchronous K-step averaging SGD (K-AVG) for nonconvex optimization, showing that delayed synchronization can outperform traditional parallel SGD and ASGD. It provides rigorous convergence bounds for fixed and diminishing step sizes, demonstrates that larger delays can be advantageous, and proves better scalability with the number of processors. Empirical results on CIFAR-10 with up to 128 GPUs confirm faster training and improved accuracy over ASGD variants, while revealing a data- and architecture-dependent optimal delay K. The work offers practical guidelines for balancing communication and computation in large-scale distributed training and shows that K-AVG can enable larger step sizes without sacrificing convergence.

Abstract

Despite their popularity, the practical performance of asynchronous stochastic gradient descent methods (ASGD) for solving large scale machine learning problems are not as good as theoretical results indicate. We adopt and analyze a synchronous K-step averaging stochastic gradient descent algorithm which we call K-AVG. We establish the convergence results of K-AVG for nonconvex objectives and explain why the K-step delay is necessary and leads to better performance than traditional parallel stochastic gradient descent which is a special case of K-AVG with $K=1$. We also show that K-AVG scales better than ASGD. Another advantage of K-AVG over ASGD is that it allows larger stepsizes. On a cluster of $128$ GPUs, K-AVG is faster than ASGD implementations and achieves better accuracies and faster convergence for \cifar dataset.

On the convergence properties of a $K$-step averaging stochastic gradient descent algorithm for nonconvex optimization

TL;DR

The paper analyzes a synchronous K-step averaging SGD (K-AVG) for nonconvex optimization, showing that delayed synchronization can outperform traditional parallel SGD and ASGD. It provides rigorous convergence bounds for fixed and diminishing step sizes, demonstrates that larger delays can be advantageous, and proves better scalability with the number of processors. Empirical results on CIFAR-10 with up to 128 GPUs confirm faster training and improved accuracy over ASGD variants, while revealing a data- and architecture-dependent optimal delay K. The work offers practical guidelines for balancing communication and computation in large-scale distributed training and shows that K-AVG can enable larger step sizes without sacrificing convergence.

Abstract

Despite their popularity, the practical performance of asynchronous stochastic gradient descent methods (ASGD) for solving large scale machine learning problems are not as good as theoretical results indicate. We adopt and analyze a synchronous K-step averaging stochastic gradient descent algorithm which we call K-AVG. We establish the convergence results of K-AVG for nonconvex objectives and explain why the K-step delay is necessary and leads to better performance than traditional parallel stochastic gradient descent which is a special case of K-AVG with . We also show that K-AVG scales better than ASGD. Another advantage of K-AVG over ASGD is that it allows larger stepsizes. On a cluster of GPUs, K-AVG is faster than ASGD implementations and achieves better accuracies and faster convergence for \cifar dataset.

Paper Structure

This paper contains 16 sections, 5 theorems, 49 equations, 8 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries and notations
  3. Main results
  4. Convergence of K-AVG
  5. K-AVG allows for larger stepsize than ASGD
  6. Scalability comparison of K-AVG against ASGD
  7. Optimal $K$ for convergence is not always $1$
  8. Experiments
  9. Comparison with ASGD
  10. The optimal delay in averaging for K-AVG
  11. Convergence comparison with SGD
  12. Conclusion
  13. Proofs
  14. Proof of Theorem \ref{['theorem:fixed stepsize']}
  15. Proof of Theorem \ref{['theorem:dynamicsize']}
  16. ...and 1 more sections

Key Result

Theorem 3.1

(Nonconvex objective, fixed stepsize, and fixed batch size) Suppose that Algorithm algorithm1 is run with a fixed stepsize $\gamma_n=\bar{\gamma}$, a fixed batch size $B_{n} = \bar{B}$ satisfying with some constant $0<\delta<1$. Then the expected average squared gradient norms of $F$ satisfy the following bounds for all $N\in \mathbb{N}$:

Figures (8)

  • Figure 1: Scaling with vgg
  • Figure 2: Scaling with nin
  • Figure 3: Speedup of K-AVG over ASGD implementations with vgg
  • Figure 4: Speedup of K-AVG over ASGD implementations with nin
  • Figure 5: Test accuracy with different $K$
  • ...and 3 more figures

Theorems & Definitions (10)

  • Theorem 3.1
  • Corollary 3.1
  • Theorem 3.2
  • Theorem 3.3
  • proof
  • Theorem 3.4
  • proof
  • proof
  • proof
  • proof