Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances

Jiangfei Duan, Ziang Song, Xupeng Miao, Xiaoli Xi, Dahua Lin, Harry Xu, Minjia Zhang, Zhihao Jia

TL;DR

Parcae is presented, a system that enables cheap, fast, and scalable DNN training on preemptible instances by proactively adjusting the parallelization strategy of a DNN training job to adapt to predicted resource changes before instance preemptions and allocations really happen, which significantly reduces the cost of handling these events.

Abstract

Deep neural networks (DNNs) are becoming progressively large and costly to train. This paper aims to reduce DNN training costs by leveraging preemptible instances on modern clouds, which can be allocated at a much lower price when idle but may be preempted by the cloud provider at any time. Prior work that supports DNN training on preemptive instances employs a reactive approach to handling instance preemptions and allocations after their occurrence, which only achieves limited performance and scalability. We present Parcae, a system that enables cheap, fast, and scalable DNN training on preemptible instances by proactively adjusting the parallelization strategy of a DNN training job to adapt to predicted resource changes before instance preemptions and allocations really happen, which significantly reduces the cost of handling these events. Parcae optimizes liveput, a novel metric that measures the expected training throughput of a DNN job under various possible preemption scenarios. Compared to existing reactive, throughput-optimized systems, Parcae's proactive, live-optimized solution considers both the throughput of a job and its robustness under preemptions. To optimize liveput, Parcae supports lightweight instance migration and uses an availability predictor to forecast future preemptions. It then uses a liveput optimizer to discover an optimal strategy to parallelize DNN training under predicted preemptions. We evaluate Parcae on a variety of DNNs and preemption traces and show that Parcae outperforms existing spot-instance DNN training systems by up to 10$\times$. More importantly, Parcae achieves near-optimal performance for training large DNNs under frequent preemptions, in which case existing approaches cannot make any progress.

Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instances

TL;DR

Parcae is presented, a system that enables cheap, fast, and scalable DNN training on preemptible instances by proactively adjusting the parallelization strategy of a DNN training job to adapt to predicted resource changes before instance preemptions and allocations really happen, which significantly reduces the cost of handling these events.

Abstract

Deep neural networks (DNNs) are becoming progressively large and costly to train. This paper aims to reduce DNN training costs by leveraging preemptible instances on modern clouds, which can be allocated at a much lower price when idle but may be preempted by the cloud provider at any time. Prior work that supports DNN training on preemptive instances employs a reactive approach to handling instance preemptions and allocations after their occurrence, which only achieves limited performance and scalability. We present Parcae, a system that enables cheap, fast, and scalable DNN training on preemptible instances by proactively adjusting the parallelization strategy of a DNN training job to adapt to predicted resource changes before instance preemptions and allocations really happen, which significantly reduces the cost of handling these events. Parcae optimizes liveput, a novel metric that measures the expected training throughput of a DNN job under various possible preemption scenarios. Compared to existing reactive, throughput-optimized systems, Parcae's proactive, live-optimized solution considers both the throughput of a job and its robustness under preemptions. To optimize liveput, Parcae supports lightweight instance migration and uses an availability predictor to forecast future preemptions. It then uses a liveput optimizer to discover an optimal strategy to parallelize DNN training under predicted preemptions. We evaluate Parcae on a variety of DNNs and preemption traces and show that Parcae outperforms existing spot-instance DNN training systems by up to 10. More importantly, Parcae achieves near-optimal performance for training large DNNs under frequent preemptions, in which case existing approaches cannot make any progress.
Paper Structure (63 sections, 6 equations, 18 figures, 5 tables, 1 algorithm)

This paper contains 63 sections, 6 equations, 18 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. Distributed DNN Training
  4. Data parallelism.
  5. Pipeline parallelism.
  6. Hybrid data and pipeline parallelism.
  7. Spot-Instance Training
  8. Liveput
  9. Definition of Liveput
  10. Comparing Liveput and Throughput
  11. Parcae Overview
  12. Availability Predictor
  13. Instance-wise Availability Unpredictability
  14. Statistical Availability Prediction
  15. Problem formulation.
  16. ...and 48 more sections

Figures (18)

  • Figure 1: Illustration of pipelined data parallelism training over on-demand and spot instance respectively. Preempted spot instances are marked with red markers. $X_j$ represents the $j$-th mini-batch of input data.
  • Figure 2: Comparing Parcae and prior work for training GPT-2 gpt2 on 32 spot GPU instances. Note that Parcae, Bamboo, and Varuna use an identical preemption trace.
  • Figure 3: Comparing the liveput and throughput of different parallel configurations and preemption scenarios.
  • Figure 4: An overview of Parcae.
  • Figure 5: (a) Comparison of normalized L1 distance of predictive performance for ARIMA and other solutions (H$=12$, lower is better). (b) Comparison between ARIMA-predicted trace (H$= 12$, I$= 4$) and the ground truth.
  • ...and 13 more figures

Theorems & Definitions (1)

  • Definition 1: Liveput