Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training DNN Models over Heterogeneous Clusters with Optimal Performance

Chengyi Nie, Jessica Maghakian, Zhenhua Liu

TL;DR

Cannikin presents a formal framework and a practical system for near-optimal training of DNNs over heterogeneous GPU clusters by learning OptPerf and optimally distributing per-node batch sizes. It combines joint performance modeling of compute and communication with overlap-aware optimization, and introduces a gradient-noise-scale based advanced estimation and weighted gradient aggregation to preserve convergence. Through a PyTorch implementation built atop AdaptDL, Cannikin demonstrates substantial reductions in convergence time and batch processing time across diverse workloads, while keeping prediction error low and overhead minimal. The work enables efficient, scalable distributed training in real-world heterogeneous environments and can be integrated with existing adaptive-batch-size engines and resource schedulers.

Abstract

Adjusting batch sizes and adaptively tuning other hyperparameters can significantly speed up deep neural network (DNN) training. Despite the ubiquity of heterogeneous clusters, existing adaptive DNN training techniques solely consider homogeneous environments. Optimizing distributed DNN training over heterogeneous clusters is technically challenging, and directly adapting existing techniques results in low utilization and poor performance. To solve this problem, we introduce Cannikin -- a novel data-parallel distributed training system. Cannikin achieves efficient and near-optimal performance by accurately modeling the optimal system performance and predicting adaptive batch size training metrics for DNNs in heterogeneous clusters. We implemented Cannikin in PyTorch and conducted experiments over 16 GPUs in Chameleon. Empirical results show that Cannikin reduces DNN training in heterogeneous clusters by up to $52\%$ compared to the state-of-the-art adaptive training system and up to $85\%$ compared to native PyTorch DistributedDataParallel.

Training DNN Models over Heterogeneous Clusters with Optimal Performance

TL;DR

Cannikin presents a formal framework and a practical system for near-optimal training of DNNs over heterogeneous GPU clusters by learning OptPerf and optimally distributing per-node batch sizes. It combines joint performance modeling of compute and communication with overlap-aware optimization, and introduces a gradient-noise-scale based advanced estimation and weighted gradient aggregation to preserve convergence. Through a PyTorch implementation built atop AdaptDL, Cannikin demonstrates substantial reductions in convergence time and batch processing time across diverse workloads, while keeping prediction error low and overhead minimal. The work enables efficient, scalable distributed training in real-world heterogeneous environments and can be integrated with existing adaptive-batch-size engines and resource schedulers.

Abstract

Adjusting batch sizes and adaptively tuning other hyperparameters can significantly speed up deep neural network (DNN) training. Despite the ubiquity of heterogeneous clusters, existing adaptive DNN training techniques solely consider homogeneous environments. Optimizing distributed DNN training over heterogeneous clusters is technically challenging, and directly adapting existing techniques results in low utilization and poor performance. To solve this problem, we introduce Cannikin -- a novel data-parallel distributed training system. Cannikin achieves efficient and near-optimal performance by accurately modeling the optimal system performance and predicting adaptive batch size training metrics for DNNs in heterogeneous clusters. We implemented Cannikin in PyTorch and conducted experiments over 16 GPUs in Chameleon. Empirical results show that Cannikin reduces DNN training in heterogeneous clusters by up to compared to the state-of-the-art adaptive training system and up to compared to native PyTorch DistributedDataParallel.
Paper Structure (32 sections, 6 theorems, 22 equations, 10 figures, 5 tables, 1 algorithm)

This paper contains 32 sections, 6 theorems, 22 equations, 10 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Data-parallel Distributed Training
  4. Adaptive Batch Size Training
  5. The OptPerf of GPU Clusters
  6. The Definition of OptPerf
  7. Performance Modeling
  8. Computing Time
  9. Gradient Synchronization Time
  10. Computing and Communication Overlap
  11. Expression of OptPerf
  12. System Design of Cannikin
  13. Workflow of Cannikin
  14. OptPerf Optimizer
  15. Optimized Gradient Aggregation
  16. ...and 17 more sections

Key Result

theorem 1

$\mathcal{G} = \sum_{i\in\mathcal{N}}w^{\mathcal{G}}_i \mathcal{G}_i$ and $\mathcal{S} = \sum_{i\in\mathcal{N}}w^{\mathcal{S}}_i \mathcal{S}_i$ are minimum variance, unbiased linear estimators of $|G|^2$ and $\text{tr}(\Sigma)$ when: where $\mathbf{1}$ is an $n$-dimensional column vector of ones and both $A_{\mathcal{G}}$ and $A_{\mathcal{S}}$ are $n\times n$ matrices with respective entries $a_{

Figures (10)

  • Figure 1: A node running in the computing-bottleneck situation. PU, DL, FP, and BP are the parameter update, data loading, forward propagation, and backpropagation.
  • Figure 2: A communication-bottleneck node.
  • Figure 3: An example of the general case, where GPU 0 and GPU 1 are communication-bottleneck, GPU 2 and GPU 3 are computing-bottleneck. FT is the abbreviation of fixed time.
  • Figure 4: The overall workflow of Cannikin.
  • Figure 5: (a) shows the total batch size selected by Cannikin and AdaptDL in a heterogeneous cluster for the training of CIFAR10. In most epochs, Cannikin trains with a larger batch size compared with AdaptDL due to the throughput improvement. (b) shows with the same training epochs, Cannikin achieves the same accuracy compared with AdaptDL. (c) shows the accuracy in the convergence process of Cannikin and AdaptDL.
  • ...and 5 more figures

Theorems & Definitions (6)

  • theorem 1
  • lemma 1
  • lemma 2
  • lemma 3
  • lemma 4
  • lemma 5