FastPersist: Accelerating Model Checkpointing in Deep Learning

TL;DR

The proposed FastPersist combines three novel techniques: NVMe optimizations for faster checkpoint writes to SSDs, efficient write parallelism using the available SSDs in training environments, and overlapping checkpointing with independent training computations.

Abstract

Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead.

Figures (15)

• Figure 1: Impact of data parallelism on training time of (a) dense and (b) sparse models, on up to 128 V100-32GB GPUs.
• Figure 2: Model checkpointing with torch.save().
• Figure 3: Data dependencies in training iterations.
• Figure 4: Comparing (a) baseline checkpointing against FastPersist using two training iterations and DP=2. FastPersist improves checkpointing efficiency with three techniques: (b) NVMe-based acceleration, (c) parallelism, and (d) pipelining.
• Figure 5: Writing tensors $T_i$ and $T_{i+1}$ from accelerator memory to NVMe. Instead of (a) serializing the two data transfers, (b) FastPersist overlaps them by double-buffering the DRAM.