QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding

TL;DR

QSGD tackles the bottleneck of communicating gradient updates in distributed SGD by combining stochastic gradient quantization with efficient coding, yielding unbiased gradient estimates with controllable variance and substantial communication reductions. It delivers convergence guarantees for convex and certain non-convex settings and extends to a variance-reduced variant (QSVRG), with precise trade-offs between bit-rate and convergence. Empirically, QSGD achieves large communication savings across deep networks and speech models with minimal or no loss in accuracy, enabling faster training on multi-GPU systems. The work also provides an open-source CNTK implementation and demonstrates practical viability for large-scale neural network training.

Abstract

Parallel implementations of stochastic gradient descent (SGD) have received significant research attention, thanks to excellent scalability properties of this algorithm, and to its efficiency in the context of training deep neural networks. A fundamental barrier for parallelizing large-scale SGD is the fact that the cost of communicating the gradient updates between nodes can be very large. Consequently, lossy compression heuristics have been proposed, by which nodes only communicate quantized gradients. Although effective in practice, these heuristics do not always provably converge, and it is not clear whether they are optimal. In this paper, we propose Quantized SGD (QSGD), a family of compression schemes which allow the compression of gradient updates at each node, while guaranteeing convergence under standard assumptions. QSGD allows the user to trade off compression and convergence time: it can communicate a sublinear number of bits per iteration in the model dimension, and can achieve asymptotically optimal communication cost. We complement our theoretical results with empirical data, showing that QSGD can significantly reduce communication cost, while being competitive with standard uncompressed techniques on a variety of real tasks. In particular, experiments show that gradient quantization applied to training of deep neural networks for image classification and automated speech recognition can lead to significant reductions in communication cost, and end-to-end training time. For instance, on 16 GPUs, we are able to train a ResNet-152 network on ImageNet 1.8x faster to full accuracy. Of note, we show that there exist generic parameter settings under which all known network architectures preserve or slightly improve their full accuracy when using quantization.

Figures (5)

• Figure 1: Parallel SGD Algorithm.
• Figure 2: Breakdown of communication versus computation for various neural networks, on $2, 4, 8, 16$ GPUs, for full 32-bit precision versus QSGD 4-bit. Each bar represents the total time for an epoch under standard parameters. Epoch time is broken down into communication (bottom, solid) and computation (top, transparent). Although epoch time diminishes as we parallelize, the proportion of communication increases.
• Figure 3: Accuracy numbers for different networks. Light blue lines represent 32-bit accuracy.
• Figure 4: Breakdown of communication versus computation for various neural networks, on $2, 4, 8, 16$ GPUs, for full 32-bit precision versus 1BitSGD versus QSGD 2-bit and 4-bit. Each bar represents the total time for an epoch under standard parameters. Epoch time is broken down into communication (bottom, solid color) and computation (top, transparent color). Notice that, although epoch time usually diminishes as we increase parallelism, the proportion of communication cost increases.
• Figure 5: Accuracy numbers for different networks. The red lines in (a) represent final 32-bit accuracy.

Theorems & Definitions (38)

• Definition 2.1
• Theorem 2.1: Bubeck, Theorem 6.3
• Corollary 2.2
• Lemma 3.1
• Theorem 3.2
• Corollary 3.3
• Theorem 3.4: Smooth Convex QSGD
• Theorem 3.5: QSGD for smooth non-convex optimization
• Theorem 3.6
• Definition A.1