Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

Amey Agrawal, Sameer Reddy, Satwik Bhattamishra, Venkata Prabhakara Sarath Nookala, Vidushi Vashishth, Kexin Rong, Alexey Tumanov

TL;DR

The experiments show that Inshrinkerator consistently achieves a better tradeoff between accuracy and compression ratio compared to prior works, enabling a compression ratio up to 39x and withstanding up to 10 restores with negligible accuracy impact in fault-tolerant training.

Abstract

With the increase in the scale of Deep Learning (DL) training workloads in terms of compute resources and time consumption, the likelihood of encountering in-training failures rises substantially, leading to lost work and resource wastage. Such failures are typically offset by a checkpointing mechanism, which comes at the cost of storage and network bandwidth overhead. State-of-the-art approaches involve lossy model compression mechanisms, which induce a tradeoff between the resulting model quality (accuracy) and compression ratio. Delta compression is then used to further reduce the overhead by only storing the difference between consecutive checkpoints. We make a key enabling observation that the sensitivity of model weights to compression varies during training, and different weights benefit from different quantization levels (ranging from retaining full precision to pruning). We propose (1) a non-uniform quantization scheme that leverages this variation, (2) an efficient search mechanism that dynamically finds the best quantization configurations, and (3) a quantization-aware delta compression mechanism that rearranges weights to minimize checkpoint differences, thereby maximizing compression. We instantiate these contributions in Inshrinkerator - a framework for DL workload checkpoint compression. Our experiments show that Inshrinkerator consistently achieves a better tradeoff between accuracy and compression ratios compared to prior works, enabling a compression ratio up to 39x and withstanding up to 10 restores with negligible accuracy impact for fault-tolerant training. Inshrinkerator achieves at least an order of magnitude reduction in checkpoint storage overhead for training failure recovery as well as transfer learning use cases without any loss of accuracy.

Inshrinkerator: Compressing Deep Learning Training Checkpoints via Dynamic Quantization

TL;DR

The experiments show that Inshrinkerator consistently achieves a better tradeoff between accuracy and compression ratio compared to prior works, enabling a compression ratio up to 39x and withstanding up to 10 restores with negligible accuracy impact in fault-tolerant training.

Abstract

With the increase in the scale of Deep Learning (DL) training workloads in terms of compute resources and time consumption, the likelihood of encountering in-training failures rises substantially, leading to lost work and resource wastage. Such failures are typically offset by a checkpointing mechanism, which comes at the cost of storage and network bandwidth overhead. State-of-the-art approaches involve lossy model compression mechanisms, which induce a tradeoff between the resulting model quality (accuracy) and compression ratio. Delta compression is then used to further reduce the overhead by only storing the difference between consecutive checkpoints. We make a key enabling observation that the sensitivity of model weights to compression varies during training, and different weights benefit from different quantization levels (ranging from retaining full precision to pruning). We propose (1) a non-uniform quantization scheme that leverages this variation, (2) an efficient search mechanism that dynamically finds the best quantization configurations, and (3) a quantization-aware delta compression mechanism that rearranges weights to minimize checkpoint differences, thereby maximizing compression. We instantiate these contributions in Inshrinkerator - a framework for DL workload checkpoint compression. Our experiments show that Inshrinkerator consistently achieves a better tradeoff between accuracy and compression ratios compared to prior works, enabling a compression ratio up to 39x and withstanding up to 10 restores with negligible accuracy impact for fault-tolerant training. Inshrinkerator achieves at least an order of magnitude reduction in checkpoint storage overhead for training failure recovery as well as transfer learning use cases without any loss of accuracy.
Paper Structure (30 sections, 3 theorems, 7 equations, 11 figures, 12 tables, 6 algorithms)

This paper contains 30 sections, 3 theorems, 7 equations, 11 figures, 12 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Background and Use Cases
  3. Overview
  4. Quantizer
  5. Metrics for Parameter Ranking
  6. Parameter Partitioning
  7. Efficient Parameter Partitioning via Quantile Sketches
  8. Non-uniform Quantization via Approximate K-Means Clustering
  9. Efficient Sensitivity Computation
  10. Quantization Config Manager
  11. Delta Encoder
  12. Implementation
  13. Evaluation Setup
  14. Evaluation Scenarios, Models and Datasets
  15. Baseline Systems
  16. ...and 15 more sections

Key Result

Proposition A.1

Given any two sets $X, \tilde{X}$ of $n$ points as defined above such that $|x_i - \tilde{x}_{i}| \leq \alpha x_i$$\forall i$. For any $k \geq 1$, the difference between the optimal loss of clusterings over $X$ and $\tilde{X}$ are bounded as,

Figures (11)

  • Figure 1: Inshrinkerator provides better accuracy-storage tradeoffs compared to baselines for ResNet152 training.
  • Figure 2: High-level workflow of a checkpoint compression and saving request, and resumption of training on failure from a saved compressed checkpoint in Inshrinkerator.
  • Figure 3: A high-level illustration of the quantization process in Inshrinkerator.
  • Figure 4: Model parameters are partitioned into three groups for pruning, protection, and quantization by jointly evaluating magnitude and sensitivity scores.
  • Figure 5: DDSketch's log space transformation creates a histogram with value-dependent bucket width.
  • ...and 6 more figures

Theorems & Definitions (6)

  • Proposition A.1
  • Remark A.1
  • Proposition A.2
  • Remark A.2
  • Definition A.1
  • Lemma A.1