Communication-Efficient Distributed Learning with Local Immediate Error Compensation

TL;DR

This work addresses the communication bottleneck in distributed stochastic optimization by introducing LIEC-SGD, a gradient-compression method that employs bidirectional compression together with local immediate error compensation while maintaining a global server-side error variable. The algorithm updates the local models with immediate compensation, enabling faster convergence and reduced communication compared to prior unidirectional or slower bidirectional methods. The authors provide non-convex convergence guarantees under a $\delta$-contraction compressor and demonstrate linear speedup behavior, along with empirical results on CIFAR-10/100 and Tiny ImageNet showing improved accuracy and shorter training times. The findings suggest that bidirectional compression, when paired with prompt error compensation, can achieve dual benefits of faster convergence and reduced communication in large-scale distributed learning.

Abstract

Gradient compression with error compensation has attracted significant attention with the target of reducing the heavy communication overhead in distributed learning. However, existing compression methods either perform only unidirectional compression in one iteration with higher communication cost, or bidirectional compression with slower convergence rate. In this work, we propose the Local Immediate Error Compensated SGD (LIEC-SGD) optimization algorithm to break the above bottlenecks based on bidirectional compression and carefully designed compensation approaches. Specifically, the bidirectional compression technique is to reduce the communication cost, and the compensation technique compensates the local compression error to the model update immediately while only maintaining the global error variable on the server throughout the iterations to boost its efficacy. Theoretically, we prove that LIEC-SGD is superior to previous works in either the convergence rate or the communication cost, which indicates that LIEC-SGD could inherit the dual advantages from unidirectional compression and bidirectional compression. Finally, experiments of training deep neural networks validate the effectiveness of the proposed LIEC-SGD algorithm.

Figures (11)

• Figure 1: Workflow of LIEC-SGD. When local immediate error-compensation is implemented, the stochastic gradient is compressed and the compression error is cached on the worker. The server gathers and averages the compressed gradients and compensates the global error to it. The result is then compressed by the server and broadcast to the worker. Each worker compensates the cached compression error to the returned value to update the model.
• Figure 2: Training ResNet18 on CIFAR-10 with operator SignSGD. (a): Test accuracy w.r.t. epochs. (b): Test accuracy w.r.t. wall-clock time. (c): Test accuracy w.r.t. communication cost.
• Figure 3: Training ResNet18 on CIFAR-10 with operator Blockwise-SignSGD. (a): Test accuracy w.r.t. epochs. (b): Test accuracy w.r.t. wall-clock time. (c): Test accuracy w.r.t. communication cost.
• Figure 4: Using LIEC-SGD to train ResNet18 on CIFAR-10 with different number of workers. (a): Training loss w.r.t. epochs. (b): Test accuracy w.r.t. epochs. (c): Test accuracy w.r.t. wall-clock time.
• Figure 5: Average time cost per epoch when training CIFAR-10.

Theorems & Definitions (21)

• Definition 1: $\delta$-contraction operator
• Definition 2: top-k
• Definition 3: random-k
• Definition 4: SignSGD
• Definition 5: Blockwise-SignSGD
• Theorem 1
• Remark 1
• Corollary 1
• Remark 2
• Remark 3