Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Truncated Non-Uniform Quantization for Distributed SGD

Guangfeng Yan, Tan Li, Yuanzhang Xiao, Congduan Li, Linqi Song

TL;DR

Truncated Non-Uniform Quantization for Distributed SGD addresses the communication bottleneck in distributed SGD by introducing a two-stage compressor that first truncates gradients to curb long-tail noise and then applies a non-uniform quantizer tailored to the gradient distribution. The authors derive a convergence bound of the form $\frac{1}{T}\sum_{t=0}^{T-1} \|\nabla F(\bm{\theta}_t)\|^2 \le \mathcal{E}_{DSGD} + \mathcal{E}_{TQ}$ and provide closed-form optimal parameters under Laplace gradient assumptions: $\alpha^*$ and $\lambda_s(g)$, yielding $\mathcal{E}_{TQ} = \frac{27 d \gamma^2}{N (s+\frac{3\sqrt{6}}{2})^2}$. The work also contrasts TNQSGD with alternative quantization strategies, showing improved convergence under fixed communication budgets, and validates the approach with MNIST experiments where TNQSGD achieves higher accuracy than competing schemes at the same bit budget.

Abstract

To address the communication bottleneck challenge in distributed learning, our work introduces a novel two-stage quantization strategy designed to enhance the communication efficiency of distributed Stochastic Gradient Descent (SGD). The proposed method initially employs truncation to mitigate the impact of long-tail noise, followed by a non-uniform quantization of the post-truncation gradients based on their statistical characteristics. We provide a comprehensive convergence analysis of the quantized distributed SGD, establishing theoretical guarantees for its performance. Furthermore, by minimizing the convergence error, we derive optimal closed-form solutions for the truncation threshold and non-uniform quantization levels under given communication constraints. Both theoretical insights and extensive experimental evaluations demonstrate that our proposed algorithm outperforms existing quantization schemes, striking a superior balance between communication efficiency and convergence performance.

Truncated Non-Uniform Quantization for Distributed SGD

TL;DR

Truncated Non-Uniform Quantization for Distributed SGD addresses the communication bottleneck in distributed SGD by introducing a two-stage compressor that first truncates gradients to curb long-tail noise and then applies a non-uniform quantizer tailored to the gradient distribution. The authors derive a convergence bound of the form and provide closed-form optimal parameters under Laplace gradient assumptions: and , yielding . The work also contrasts TNQSGD with alternative quantization strategies, showing improved convergence under fixed communication budgets, and validates the approach with MNIST experiments where TNQSGD achieves higher accuracy than competing schemes at the same bit budget.

Abstract

To address the communication bottleneck challenge in distributed learning, our work introduces a novel two-stage quantization strategy designed to enhance the communication efficiency of distributed Stochastic Gradient Descent (SGD). The proposed method initially employs truncation to mitigate the impact of long-tail noise, followed by a non-uniform quantization of the post-truncation gradients based on their statistical characteristics. We provide a comprehensive convergence analysis of the quantized distributed SGD, establishing theoretical guarantees for its performance. Furthermore, by minimizing the convergence error, we derive optimal closed-form solutions for the truncation threshold and non-uniform quantization levels under given communication constraints. Both theoretical insights and extensive experimental evaluations demonstrate that our proposed algorithm outperforms existing quantization schemes, striking a superior balance between communication efficiency and convergence performance.
Paper Structure (14 sections, 4 theorems, 38 equations, 3 figures, 1 algorithm)

This paper contains 14 sections, 4 theorems, 38 equations, 3 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Truncated Non-Uniform Quantizer for Distributed SGD
  4. Optimal Parameters Design For Truncated Non-Uniform Quantizer
  5. Optimal Parameters for Any Gradient Distribution
  6. Optimal Parameters for Laplace Gradient Distribution
  7. Relationship With Other Quantization Scheme
  8. Experiments
  9. Conclusion
  10. Appendix
  11. Proof of Lemma 1
  12. Proof of Lemma 2
  13. Proof of Lemma 3
  14. Proof of Theorem 1

Key Result

Lemma 1

For a truncated gradient element $g\in [a_1,a_2]$ with probability density function $p_g(\cdot)$, given the quantization points $\mathcal{L}=\{l_0, l_1,...,l_s\}$, the nonuniform stochastic quantization satisfies: and where $P_k = \int_{l_{k-1}}^{l_k}p_g(x)\mathrm{d}x$ and $|\Delta_k| = l_k - l_{k-1}$.

Figures (3)

  • Figure 1: Truncated Non-Uniform Quantizer (With truncation threshold $[-\alpha, \alpha]$ and quantization bit $b=3$ and quantization level $s=7$.)
  • Figure 2: Model performance of different algorithms.
  • Figure 3: Communication-learning tradeoff of different algorithms.

Theorems & Definitions (4)

  • Lemma 1: Unbiasness and Bounded Variance
  • Lemma 2
  • Theorem 1
  • Lemma 3