Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Global Momentum Compression for Sparse Communication in Distributed Learning

Chang-Wei Shi, Shen-Yi Zhao, Yin-Peng Xie, Hao Gao, Wu-Jun Li

TL;DR

Global Momentum Compression (GMC) introduces a global momentum term into sparse communication for distributed momentum SGD, addressing the bias issues of local momentum under sparsification. GMC+ extends this framework with detached error feedback to tolerate highly aggressive compressors like RBGS, while preserving convergence guarantees. Theoretical results show GMC and GMC+ converge at a rate of $\mathcal{O}(1/\sqrt{KT})$, achieving linear speedup in the number of workers, and empirical experiments on CIFAR-10/100 with ResNet20 and ViT demonstrate improved accuracy and faster convergence, especially under non-IID data distributions. The work provides a practical, communication-efficient approach for large-scale distributed training with theoretical guarantees and strong empirical performance.

Abstract

With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep models. Due to the latency and limited bandwidth of the network, communication has become the bottleneck of distributed learning. Communication compression with sparsified gradient, abbreviated as \emph{sparse communication}, has been widely employed to reduce communication cost. All existing works about sparse communication in DMSGD employ local momentum, in which the momentum only accumulates stochastic gradients computed by each worker locally. In this paper, we propose a novel method, called \emph{\underline{g}}lobal \emph{\underline{m}}omentum \emph{\underline{c}}ompression~(GMC), for sparse communication. Different from existing works that utilize local momentum, GMC utilizes global momentum. Furthermore, to enhance the convergence performance when using more aggressive sparsification compressors (e.g., RBGS), we extend GMC to GMC+. We theoretically prove the convergence of GMC and GMC+. To the best of our knowledge, this is the first work that introduces global momentum for sparse communication in distributed learning. Empirical results demonstrate that, compared with the local momentum counterparts, our GMC and GMC+ can achieve higher test accuracy and exhibit faster convergence, especially under non-IID data distribution.

Global Momentum Compression for Sparse Communication in Distributed Learning

TL;DR

Global Momentum Compression (GMC) introduces a global momentum term into sparse communication for distributed momentum SGD, addressing the bias issues of local momentum under sparsification. GMC+ extends this framework with detached error feedback to tolerate highly aggressive compressors like RBGS, while preserving convergence guarantees. Theoretical results show GMC and GMC+ converge at a rate of , achieving linear speedup in the number of workers, and empirical experiments on CIFAR-10/100 with ResNet20 and ViT demonstrate improved accuracy and faster convergence, especially under non-IID data distributions. The work provides a practical, communication-efficient approach for large-scale distributed training with theoretical guarantees and strong empirical performance.

Abstract

With the rapid growth of data, distributed momentum stochastic gradient descent~(DMSGD) has been widely used in distributed learning, especially for training large-scale deep models. Due to the latency and limited bandwidth of the network, communication has become the bottleneck of distributed learning. Communication compression with sparsified gradient, abbreviated as \emph{sparse communication}, has been widely employed to reduce communication cost. All existing works about sparse communication in DMSGD employ local momentum, in which the momentum only accumulates stochastic gradients computed by each worker locally. In this paper, we propose a novel method, called \emph{\underline{g}}lobal \emph{\underline{m}}omentum \emph{\underline{c}}ompression~(GMC), for sparse communication. Different from existing works that utilize local momentum, GMC utilizes global momentum. Furthermore, to enhance the convergence performance when using more aggressive sparsification compressors (e.g., RBGS), we extend GMC to GMC+. We theoretically prove the convergence of GMC and GMC+. To the best of our knowledge, this is the first work that introduces global momentum for sparse communication in distributed learning. Empirical results demonstrate that, compared with the local momentum counterparts, our GMC and GMC+ can achieve higher test accuracy and exhibit faster convergence, especially under non-IID data distribution.

Paper Structure

This paper contains 25 sections, 8 theorems, 65 equations, 7 figures, 4 tables, 4 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Notations
  4. Problem Formulation
  5. Distributed Momentum SGD
  6. Global Momentum Compression
  7. Framework of GMC
  8. Benefit of Global Momentum
  9. Convergence of GMC
  10. GMC+
  11. Experiments
  12. Results of GMC
  13. Results of GMC+
  14. Conclusion
  15. Other Ways for Combining Momentum and Error Feedback
  16. ...and 10 more sections

Key Result

Lemma 3

Let ${\bf z}_t \triangleq {\bf w}_{t}+\frac{\beta}{1-\beta}({\bf w}_{t} - {\bf w}_{t-1}) - \frac{\eta}{1-\beta} \bar{{\bf e}}_{t}$, then we have:

Figures (7)

  • Figure 1: Comparison of optimization trajectories of different methods.
  • Figure 2: Comparision of the distances to the global optimal point of different methods.
  • Figure 3: Training curves of different methods under IID data distribution.
  • Figure 4: Training curves of different methods under non-IID data distribution.
  • Figure 5: Training curves of DEF-A and GMC$+$ under IID data distribution.
  • ...and 2 more figures

Theorems & Definitions (14)

  • Remark 1
  • Remark 2
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • Theorem 6
  • Remark 7
  • Remark 8
  • Lemma 9
  • Lemma 10
  • ...and 4 more