Empowering Distributed Training with Sparsity-driven Data Synchronization

TL;DR

This work targets the gradient synchronization bottleneck in distributed deep learning by exploiting tensor sparsity. It first characterizes sparse gradients across popular models and formalizes a four-dimensional design space for sparse synchronization, proving that near-optimal schemes exist (Balanced Parallelism or Hierarchical Centralization). Leveraging these insights, it introduces Zen, a data-independent system employing a hierarchical hashing algorithm and a hash bitmap encoding to achieve load-balanced, low-overhead synchronization of sparse gradients. On diverse workloads, Zen delivers up to 5.09x reduction in communication time and up to 2.48x gains in training throughput, outperforming state-of-the-art sparse-synchronization methods. The approach promises practical impact for scalable distributed training of sparse models and large language models by tightly aligning synchronization strategy with observed sparsity patterns.

Abstract

Distributed training is the de facto standard to scale up the training of deep learning models with multiple GPUs. Its performance bottleneck lies in communications for gradient synchronization. Although high tensor sparsity is widely observed, the optimal communication scheme to fully leverage sparsity is still missing. This paper aims to bridge this gap. We first analyze the characteristics of sparse tensors in popular models to understand the fundamentals of sparsity. We then systematically explore the design space of communication schemes for sparse tensors and find the optimal ones. These findings give a new understanding and inspire us to develop a holistic gradient synchronization system called Zen for sparse tensors. We demonstrate that Zen can achieve up to 5.09x speedup in communication time and up to $2.48\times$ speedup in training throughput compared to the state-of-the-art methods.

Figures (15)

• Figure 1: Zen System Overview.
• Figure 2: The characteristics of sparse tensors in DL models. (a) shows that the overlap ratio of sparse tensors varies; (b) shows that tensors have higher density after aggregation.
• Figure 3: The distribution of non-zero gradients is skewed. (a) The heatmap of non-zero gradients distribution; (b) The skewness ratio.
• Figure 4: An illustration of three communication patterns with four GPUs. GPU $P_3$ aggregates the data from all GPUs.
• Figure 5: An Illustration of two aggregation patterns with Hierarchy. The gradients on each GPU are from the same parameter and 4.7 is the final aggregated result.

Theorems & Definitions (21)

• Definition 1: Dense tensor
• Definition 2: Sparse tensor
• Definition 3: The overlap ratio
• Definition 4: The densification ratio
• Definition 5: The skewness ratio
• Definition 6: Balanced Parallelism
• Definition 7: Hierarchical Centralization
• Theorem 1: Optimal schemes
• proof
• Lemma 1