Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections

Marcel Wagenländer, Guo Li, Bo Zhao, Luo Mai, Peter Pietzuch

TL;DR

Tenplex tackles the challenge of dynamic GPU changes for DL jobs that employ multi-dimensional parallelism. It introduces a parallelizable tensor collection (PTC) to externalize and unify dataset and model state, and a State Transformer that computes and executes reconfiguration plans with minimal data movement. The approach enables arbitrary reconfiguration across data, tensor, and pipeline parallelism, supported by an in-memory Tensor Store and distributed state transformations, and demonstrates substantial reductions in reconfiguration overhead and improvements in throughput and resilience. This work provides a practical, architecture-agnostic pathway to make long-running DL workloads adaptable to elasticity, redeployment, and failures in large GPU clusters.

Abstract

Deep learning (DL) jobs use multi-dimensional parallelism, i.e. combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. In particular, they cannot change the multi-dimensional parallelism of an already-running job in an efficient and model-independent way. We describe Scalai, a state management library for DL systems that enables jobs to change their parallelism dynamically after the GPU allocation is updated at runtime. Scalai achieves this through a new abstraction, a parallelizable tensor collection (PTC), that externalizes the job state during training. After a GPU change, Scalai uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to DL workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration. For efficiency, Scalai executes PTC transformations in parallel with minimum data movement between workers. Our experiments show that Scalai enables DL jobs to support dynamic parallelization with low overhead.

Tenplex: Dynamic Parallelism for Deep Learning using Parallelizable Tensor Collections

TL;DR

Tenplex tackles the challenge of dynamic GPU changes for DL jobs that employ multi-dimensional parallelism. It introduces a parallelizable tensor collection (PTC) to externalize and unify dataset and model state, and a State Transformer that computes and executes reconfiguration plans with minimal data movement. The approach enables arbitrary reconfiguration across data, tensor, and pipeline parallelism, supported by an in-memory Tensor Store and distributed state transformations, and demonstrates substantial reductions in reconfiguration overhead and improvements in throughput and resilience. This work provides a practical, architecture-agnostic pathway to make long-running DL workloads adaptable to elasticity, redeployment, and failures in large GPU clusters.

Abstract

Deep learning (DL) jobs use multi-dimensional parallelism, i.e. combining data, model, and pipeline parallelism, to use large GPU clusters efficiently. Long-running jobs may experience changes to their GPU allocation: (i) resource elasticity during training adds or removes GPUs; (ii) hardware maintenance may require redeployment on different GPUs; and (iii) GPU failures force jobs to run with fewer devices. Current DL frameworks tie jobs to a set of GPUs and thus lack support for these scenarios. In particular, they cannot change the multi-dimensional parallelism of an already-running job in an efficient and model-independent way. We describe Scalai, a state management library for DL systems that enables jobs to change their parallelism dynamically after the GPU allocation is updated at runtime. Scalai achieves this through a new abstraction, a parallelizable tensor collection (PTC), that externalizes the job state during training. After a GPU change, Scalai uses the PTC to transform the job state: the PTC repartitions the dataset state under data parallelism and exposes it to DL workers through a virtual file system; and the PTC obtains the model state as partitioned checkpoints and transforms them to reflect the new parallelization configuration. For efficiency, Scalai executes PTC transformations in parallel with minimum data movement between workers. Our experiments show that Scalai enables DL jobs to support dynamic parallelization with low overhead.
Paper Structure (27 sections, 1 equation, 16 figures, 1 table, 1 algorithm)

This paper contains 27 sections, 1 equation, 16 figures, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Resources Changes in DL Training
  3. Deep learning jobs with parallelism
  4. Need for dynamic resource changes
  5. Challenges when changing GPU resources
  6. Current approaches
  7. Tenplex Design
  8. Parallelizable Tensor Collection
  9. PTC overview
  10. Reconfiguration plan
  11. Expanding to new parallelism strategies
  12. Tenplex Architecture
  13. State Transformer
  14. Tensor Store
  15. Fault tolerance
  16. ...and 12 more sections

Figures (16)

  • Figure 1: Training deep learning (DL) jobs with multi-dimensional parallelism on a shared GPU cluster
  • Figure 2: Impact of GPU change on training convergence (Changing GPUs from 2 to 4 with GPT-3 and MNIST)
  • Figure 3: Performance impact of different parallelization configurations on 16 GPUs
  • Figure 4: Approaches for dynamic resource changes in DL jobs
  • Figure 5: Tenplex design
  • ...and 11 more figures