Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

TL;DR

This survey addresses the communication bottleneck in distributed deep learning by presenting a four-dimensional taxonomy (synchronization, system architectures, compression, and scheduling) and by benchmarking a wide range of algorithms. It synthesizes theory and practice across BSP-SGD, SSP, ASP-SGD, Local-SGD, PS, All-Reduce, and Gossip frameworks, detailing quantization and sparsification techniques, scheduling strategies, convergence analyses, and auxiliary tricks. Key contributions include a unified benchmark framework, comprehensive convergence bounds across architectures, and practical guidance on selecting methods for given hardware and network conditions. The work is practically significant for optimizing training efficiency on large-scale models and datasets, including potential extensions to foundation-model training and adaptive compression schemes.

Abstract

Distributed deep learning (DL) has become prevalent in recent years to reduce training time by leveraging multiple computing devices (e.g., GPUs/TPUs) due to larger models and datasets. However, system scalability is limited by communication becoming the performance bottleneck. Addressing this communication issue has become a prominent research topic. In this paper, we provide a comprehensive survey of the communication-efficient distributed training algorithms, focusing on both system-level and algorithmic-level optimizations. We first propose a taxonomy of data-parallel distributed training algorithms that incorporates four primary dimensions: communication synchronization, system architectures, compression techniques, and parallelism of communication and computing tasks. We then investigate state-of-the-art studies that address problems in these four dimensions. We also compare the convergence rates of different algorithms to understand their convergence speed. Additionally, we conduct extensive experiments to empirically compare the convergence performance of various mainstream distributed training algorithms. Based on our system-level communication cost analysis, theoretical and experimental convergence speed comparison, we provide readers with an understanding of which algorithms are more efficient under specific distributed environments. Our research also extrapolates potential directions for further optimizations.

Figures (5)

• Figure 1: The trends of GPU and neural networks.
• Figure 2: Overview of data-parallel distributed deep learning.
• Figure 3: Comparison of training with single device and multiple devices under different communication synchronization with the PS architecture.
• Figure 4: Three different system architectures for model/gradient aggregation. In (a) and (b), workers synchronize trained models and conduct local training with the synchronized model. In (c), workers communicate model parameters with some neighbors, and start local training with heterogeneous models parameters.
• Figure 5: Comparison of Quantization and Sparsification.