Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PowerAI DDL

Minsik Cho, Ulrich Finkler, Sameer Kumar, David Kung, Vaibhav Saxena, Dheeraj Sreedhar

TL;DR

PowerAI DDL introduces a software-hardware co-optimized distributed DL system that achieves near-linear scaling up to 256 GPUs via a topology-aware multi-ring communication library, integrated into TensorFlow, Caffe, and Torch. It demonstrates competitive accuracy and substantially reduced training times on ResNet-50 (1K) and ResNet-101 (22K) for Imagenet, with favorable communication overhead versus prior work. The approach hinges on multi-dimensional ring reductions and gradient synchronization primitives (ReduceScatter/AllGather) tuned to network topology, leveraging IBM Spectrum MPI on Power8 hardware. The work highlights significant practical impact for rapid model iteration and calls for resilience and automation to enable broader, cloud-based deployment.

Abstract

As deep neural networks become more complex and input datasets grow larger, it can take days or even weeks to train a deep neural network to the desired accuracy. Therefore, distributed Deep Learning at a massive scale is a critical capability, since it offers the potential to reduce the training time from weeks to hours. In this paper, we present a software-hardware co-optimized distributed Deep Learning system that can achieve near-linear scaling up to hundreds of GPUs. The core algorithm is a multi-ring communication pattern that provides a good tradeoff between latency and bandwidth and adapts to a variety of system configurations. The communication algorithm is implemented as a library for easy use. This library has been integrated into Tensorflow, Caffe, and Torch. We train Resnet-101 on Imagenet 22K with 64 IBM Power8 S822LC servers (256 GPUs) in about 7 hours to an accuracy of 33.8 % validation accuracy. Microsoft's ADAM and Google's DistBelief results did not reach 30 % validation accuracy for Imagenet 22K. Compared to Facebook AI Research's recent paper on 256 GPU training, we use a different communication algorithm, and our combined software and hardware system offers better communication overhead for Resnet-50. A PowerAI DDL enabled version of Torch completed 90 epochs of training on Resnet 50 for 1K classes in 50 minutes using 64 IBM Power8 S822LC servers (256 GPUs).

PowerAI DDL

TL;DR

PowerAI DDL introduces a software-hardware co-optimized distributed DL system that achieves near-linear scaling up to 256 GPUs via a topology-aware multi-ring communication library, integrated into TensorFlow, Caffe, and Torch. It demonstrates competitive accuracy and substantially reduced training times on ResNet-50 (1K) and ResNet-101 (22K) for Imagenet, with favorable communication overhead versus prior work. The approach hinges on multi-dimensional ring reductions and gradient synchronization primitives (ReduceScatter/AllGather) tuned to network topology, leveraging IBM Spectrum MPI on Power8 hardware. The work highlights significant practical impact for rapid model iteration and calls for resilience and automation to enable broader, cloud-based deployment.

Abstract

As deep neural networks become more complex and input datasets grow larger, it can take days or even weeks to train a deep neural network to the desired accuracy. Therefore, distributed Deep Learning at a massive scale is a critical capability, since it offers the potential to reduce the training time from weeks to hours. In this paper, we present a software-hardware co-optimized distributed Deep Learning system that can achieve near-linear scaling up to hundreds of GPUs. The core algorithm is a multi-ring communication pattern that provides a good tradeoff between latency and bandwidth and adapts to a variety of system configurations. The communication algorithm is implemented as a library for easy use. This library has been integrated into Tensorflow, Caffe, and Torch. We train Resnet-101 on Imagenet 22K with 64 IBM Power8 S822LC servers (256 GPUs) in about 7 hours to an accuracy of 33.8 % validation accuracy. Microsoft's ADAM and Google's DistBelief results did not reach 30 % validation accuracy for Imagenet 22K. Compared to Facebook AI Research's recent paper on 256 GPU training, we use a different communication algorithm, and our combined software and hardware system offers better communication overhead for Resnet-50. A PowerAI DDL enabled version of Torch completed 90 epochs of training on Resnet 50 for 1K classes in 50 minutes using 64 IBM Power8 S822LC servers (256 GPUs).

Paper Structure

This paper contains 17 sections, 2 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Prior Art
  3. Distributed Deep Learning Benchmarking Methodology
  4. Scaling efficiency
  5. Accuracy and end to end training time
  6. The neural network
  7. The Deep Learning framework
  8. The GPU
  9. The communication overhead
  10. The PowerAI DDL Library
  11. PowerAI DDL objects
  12. Current implementation
  13. Topology aware communication
  14. Experimental Results
  15. Resnet-50 1K
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Communication ring implemented on a tree topology that uses all links in both directions concurrently.
  • Figure 2: Resnet-50 for 1K classes using up to 256 GPUs with Caffe.
  • Figure 3: Resnet-50 for 1K classes using up to 256 GPUs with Torch.
  • Figure 4: Resnet-101 for 22k classes using up to 256 GPUs.