Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Understanding Top-k Sparsification in Distributed Deep Learning

Shaohuai Shi, Xiaowen Chu, Ka Chun Cheung, Simon See

TL;DR

This work tackles the communication bottleneck in distributed SGD by focusing on Top$_k$ gradient sparsification with error compensation. It shows that gradient coordinates tend to follow a bell-shaped, Gaussian-like distribution during training, which motivates a tighter contraction bound for the Top$_k$ operator: $ rac{ rm{{m{u}}- ext{Top}_k({m{u}})}^2}{ rm{{m{u}}}^2} leq (1- rac{k}{d})^2$ and a related delta $ abla= rac{2kd - k^2}{d^2}$, leading to convergence guarantees that more closely explain observed performance than prior bounds. The paper then introduces Gaussian$_k$, a GPU-friendly approximate Top$_k$ that estimates the threshold using the percentile point function with $p=1- rac{k}{d}$ and gradient statistics $(bmu,bsigma)$, achieving $O(d)$ complexity while preserving accuracy. Extensive experiments across CNNs and RNNs validate that GaussianK-SGD closely matches TopK-SGD in convergence with small accuracy losses, and demonstrate substantial end-to-end speedups on a 16-GPU cluster compared to dense training and existing sparsification methods. These results provide a practical, scalable pathway for distributed training by combining tighter theory with an efficient top-$k$ selection algorithm.

Abstract

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-$k$ sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without an obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top-$k$ operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-$k$) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during the training process through extensive experiments. We then theoretically derive a tighter bound for the Top-$k$ operator. Finally, we exploit the property of gradient distribution to propose an approximate top-$k$ selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead. Codes are available at: \url{https://github.com/hclhkbu/GaussianK-SGD}.

Understanding Top-k Sparsification in Distributed Deep Learning

TL;DR

This work tackles the communication bottleneck in distributed SGD by focusing on Top gradient sparsification with error compensation. It shows that gradient coordinates tend to follow a bell-shaped, Gaussian-like distribution during training, which motivates a tighter contraction bound for the Top operator: and a related delta , leading to convergence guarantees that more closely explain observed performance than prior bounds. The paper then introduces Gaussian, a GPU-friendly approximate Top that estimates the threshold using the percentile point function with and gradient statistics , achieving complexity while preserving accuracy. Extensive experiments across CNNs and RNNs validate that GaussianK-SGD closely matches TopK-SGD in convergence with small accuracy losses, and demonstrate substantial end-to-end speedups on a 16-GPU cluster compared to dense training and existing sparsification methods. These results provide a practical, scalable pathway for distributed training by combining tighter theory with an efficient top- selection algorithm.

Abstract

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top- sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without an obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top- operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during the training process through extensive experiments. We then theoretically derive a tighter bound for the Top- operator. Finally, we exploit the property of gradient distribution to propose an approximate top- selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead. Codes are available at: \url{https://github.com/hclhkbu/GaussianK-SGD}.

Paper Structure

This paper contains 19 sections, 2 theorems, 14 equations, 11 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Study on Stochastic Gradients
  4. Gradient Distribution
  5. Theoretical Analysis and Results
  6. $\text{Gaussian}_k$: An Approximate $\text{Top}_k$ Operator
  7. Experiments
  8. Experimental Settings
  9. Numerical Results of the $\text{Top}_k$ Operator
  10. GPU Computation Efficiency of Sparsification
  11. Convergence Performance of GaussianK-SGD.
  12. End-to-end Training Scaling Efficiency of GaussianK-SGD.
  13. Conclusion
  14. Appendix
  15. Cumulative Distribution of Gradients in TopK-SGD
  16. ...and 4 more sections

Key Result

Theorem 1

Assume that ${\bm{u}} \in {\mathbb{R}}^d$ follows a bell shaped distribution and $\bm{\pi}^2$ is convex and less than the line $y=-i/d+1$, then we have Furthermore, it can be rearranged into the form that

Figures (11)

  • Figure 1: Convergence comparison between original distributed SGD (Dense-SGD), $\text{Top}_k$ sparsification (TopK-SGD) and $\text{Rand}_k$ sparsification (RandK-SGD) at 16 distributed workers on the CIFAR10 krizhevsky2010cifar and ImageNet deng2009imagenet data sets. $k=0.001d$ for TopK-SGD and RandK-SGD.
  • Figure 2: The histograms of ${\bm{u}}_t^1$ of TopK-SGD. For each model, the gradient histograms are plotted every 200 iterations from iteration 200 to 1600 (other iterations have similar shapes).
  • Figure 3: The shape of $\bm{\pi}_{(i)}^2$ with different $i$ with $d=100,000$ and $\sigma=1$.
  • Figure 4: The GPU computation time (lower is better) of $\text{Top}_k$, $\text{DGC}_k$ and $\text{Gaussian}_k$. We use the PyTorch tensor API, "tensor.topk()", for the $\text{Top}_k$ operator.
  • Figure 5: The comparison of bounds with a range of $k$.
  • ...and 6 more figures

Theorems & Definitions (2)

  • Theorem 1
  • Theorem 2