Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms

TL;DR

This paper introduces Cooperative SGD, a unified framework for analyzing communication-efficient distributed SGD that encompasses periodic-averaging, elastic-averaging, and decentralized SGD. It provides a unified nonconvex convergence analysis showing how communication period, network topology, and auxiliary variables shape the error floor and convergence rate, and it removes the need for uniformly bounded gradients. The work derives new insights, including the optimal elasticity parameter for EASGD and a comparison criterion between PASGD and D-PSGD, and uses these to design novel variants such as decentralized periodic averaging, generalized elastic averaging, and hierarchical averaging. The results offer a principled design space for faster, scalable distributed learning with controlled communication overhead, supported by theoretical guarantees and empirical demonstrations.

Abstract

Communication-efficient SGD algorithms, which allow nodes to perform local updates and periodically synchronize local models, are highly effective in improving the speed and scalability of distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies remains a largely open problem. This paper presents a unified framework called Cooperative SGD that subsumes existing communication-efficient SGD algorithms such as periodic-averaging, elastic-averaging and decentralized SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for existing algorithms. Moreover, this framework enables us to design new communication-efficient SGD algorithms that strike the best balance between reducing communication overhead and achieving fast error convergence with low error floor.

Figures (7)

• Figure 1: Illustration of communication-reduction strategies for $\tau = 4$. Blue, red, grey arrows represent gradient computation, communication among workers, and update of auxiliary variables respectively.
• Figure 2: Illustration of how the network error bound in \ref{['eqn:err_bnd']} monotonically increases with $\tau$ and $\zeta$.
• Figure 3: Experiments on CIFAR-10 with VGG-16 and 8 worker nodes. For the same learning rate, larger $\tau$ or larger $\zeta$ lead to a higher error floor at convergence. Each line corresponds to a circled point in \ref{['fig:ne']}.
• Figure 4: EASGD training on CIFAR-10 with VGG-16. Since there are 8 worker nodes and 1 auxiliary variable, the best value of $\alpha$ given by \ref{['lem:opt_a']} is $2/(m+2) = 0.2$, which performs better than the empirical choice $\alpha = 0.9/m = 0.1125$ suggested in zhang2015deep. The best choice of $\alpha$ yields the lowest training loss and the least discrepancies between workers and auxiliary variable.
• Figure 5: Decentralized periodic averaging on CIFAR-10 with VGG-16. It achieves significant speedup over pure D-PSGD and has lower training loss than pure PASGD with a large communication period.

Theorems & Definitions (24)

• Remark 1
• Theorem 1: Convergence of Cooperative SGD
• Corollary 1
• Lemma 1: Best Choice of $\alpha$
• Theorem 2: Convergence of EASGD with the best $\alpha$
• Lemma 2
• Corollary 2: Convergence of PASGD
• Corollary 3: Convergence of D-PSGD
• Definition 1: horn1990matrix
• Definition 2: horn1990matrix