Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

8-Bit Approximations for Parallelism in Deep Learning

Tim Dettmers

TL;DR

This work addresses the bottleneck of inter-device communication in large-scale deep learning by introducing 8-bit gradient and activation approximations that compress 32-bit data without measurable loss in accuracy on benchmarks spanning MNIST to ImageNet. It presents four 8-bit data representations, with a dynamic exponent and a binary-tree mantissa that improve approximation quality and combine with a sub-batch strategy to hide communication costs in both data and model parallelism. A predictive speedup model, validated against hardware benchmarks, shows substantial gains on large GPU clusters (e.g., up to ~50x with 96 GPUs) while maintaining competitive accuracy, establishing 8-bit approximations as state-of-the-art for model parallelism in CNNs. The results imply significant practical impact for scalable CNN training on very large GPU systems, motivating further hardware and algorithmic advances to exploit high-bandwidth networks and memory architectures.

Abstract

The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.

8-Bit Approximations for Parallelism in Deep Learning

TL;DR

This work addresses the bottleneck of inter-device communication in large-scale deep learning by introducing 8-bit gradient and activation approximations that compress 32-bit data without measurable loss in accuracy on benchmarks spanning MNIST to ImageNet. It presents four 8-bit data representations, with a dynamic exponent and a binary-tree mantissa that improve approximation quality and combine with a sub-batch strategy to hide communication costs in both data and model parallelism. A predictive speedup model, validated against hardware benchmarks, shows substantial gains on large GPU clusters (e.g., up to ~50x with 96 GPUs) while maintaining competitive accuracy, establishing 8-bit approximations as state-of-the-art for model parallelism in CNNs. The results imply significant practical impact for scalable CNN training on very large GPU systems, motivating further hardware and algorithmic advances to exploit high-bandwidth networks and memory architectures.

Abstract

The creation of practical deep learning data-products often requires parallelization across processors and computers to make deep learning feasible on large data sets, but bottlenecks in communication bandwidth make it difficult to attain good speedups through parallelism. Here we develop and test 8-bit approximation algorithms which make better use of the available bandwidth by compressing 32-bit gradients and nonlinear activations to 8-bit approximations. We show that these approximations do not decrease predictive performance on MNIST, CIFAR10, and ImageNet for both model and data parallelism and provide a data transfer speedup of 2x relative to 32-bit parallelism. We build a predictive model for speedups based on our experimental data, verify its validity on known speedup data, and show that we can obtain a speedup of 50x and more on a system of 96 GPUs compared to a speedup of 23x for 32-bit. We compare our data types with other methods and show that 8-bit approximations achieve state-of-the-art speedups for model parallelism. Thus 8-bit approximation is an efficient method to parallelize convolutional networks on very large systems of GPUs.

Paper Structure

This paper contains 25 sections, 7 equations, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Data parallelism
  4. Model parallelism
  5. General bottlenecks
  6. PCIe switches
  7. Bandwidth
  8. Latency
  9. Optimal parallelism for convolutional nets
  10. 8-bit approximation
  11. Designing 8-bit data types
  12. Implementation and computational performance
  13. Theoretical speedup
  14. Approximation error
  15. Comparison to other methods
  16. ...and 10 more sections

Figures (2)

  • Figure 1: Anatomy of the four different 8-bit data types. Note that the dynamic data type shown here is a specific example and the number of bits for the exponent and for the tree varies between individual floating point numbers.
  • Figure 2: Classification train and test error for the 8-bit dynamic data type used in AlexNet on the ImageNet dataset.