Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

TL;DR

This work tackles the communication bottleneck in distributed learning by introducing SASG, a method that combines adaptive aggregation with gradient sparsification and error-feedback. It proves convergence at a sublinear rate matching SGD, using a novel Lyapunov analysis that couples model updates with compression error via an auxiliary variable $\nu^t$. Theoretical results show the rate $\mathcal{O}(1/\sqrt{T})$ with a sparsification parameter $\delta=k/d$ influencing constants, and experiments on MNIST and CIFAR demonstrate substantial reductions in communication rounds and transmitted bits without sacrificing accuracy. The approach scales with the number of workers, making it practical for large-scale distributed systems, and provides a principled framework for reducing both rounds and bits in uplink communications.

Abstract

Gradient-based optimization methods implemented on distributed computing architectures are increasingly used to tackle large-scale machine learning applications. A key bottleneck in such distributed systems is the high communication overhead for exchanging information, such as stochastic gradients, between workers. The inherent causes of this bottleneck are the frequent communication rounds and the full model gradient transmission in every round. In this study, we present SASG, a communication-efficient distributed algorithm that enjoys the advantages of sparse communication and adaptive aggregated stochastic gradients. By dynamically determining the workers who need to communicate through an adaptive aggregation rule and sparsifying the transmitted information, the SASG algorithm reduces both the overhead of communication rounds and the number of communication bits in the distributed system. For the theoretical analysis, we introduce an important auxiliary variable and define a new Lyapunov function to prove that the communication-efficient algorithm is convergent. The convergence result is identical to the sublinear rate of stochastic gradient descent, and our result also reveals that SASG scales well with the number of distributed workers. Finally, experiments on training deep neural networks demonstrate that the proposed algorithm can significantly reduce communication overhead compared to previous methods.

Figures (5)

• Figure 1: SASG overview. At the $t$-th iteration, the parameter server broadcasts ${\omega}^{t}$ to all workers, and workers in $\mathcal{M}^{t}$ (marked in green) will upload the sparsified gradient information $\mathcal{T}_{k}(g_{m}^{t})$ and reset $\tau_{m}^{t+1}=1$; while workers in $\mathcal{M}_{c}^{t}$ (marked in pink) will increase the staleness by $\tau_{m}^{t+1}=\tau_{m}^{t}+1$ and upload nothing; then the server updates parameter $\omega$ via \ref{['scheme']}.
• Figure 2: Experimental results of test accuracy percentage and training loss versus communication round in three settings. All four methods are trained with the same number of epochs. Our algorithm significantly reduces the number of communication rounds required to achieve the same performance and complete the training.
• Figure 3: Experimental results of classifying CIFAR-10 with ResNet18. Figures (a) and (b) show the test accuracy versus communication time. Figure (c) records the communication time every $100$ iterations, where the learning rate decay was used at the $10,000$-th iteration.
• Figure 4: Experimental results of training ResNet18 to classify CIFAR-10 using different sparsity rates.
• Figure 5: Experimental results of training the ResNet18 model in heterogeneous data scenarios.

Theorems & Definitions (9)

• Definition 1
• Lemma 1
• Lemma 2
• Definition 2
• Lemma 3
• Lemma 4
• Theorem 1
• Remark 1
• Remark 2