Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lossless Compression for LLM Tensor Incremental Snapshots

Daniel Waddington, Cornel Constantinescu

TL;DR

The paper tackles the large data burden of checkpointing during LLM training by analyzing the structure and evolution of tensor deltas and proposing a lossless compression pipeline. It introduces Byte-Grouping with Run-Length Encoding and a block-adaptive Huffman coder (LMC), extended to a parallel variant (PLMC) to achieve high throughput. Empirical results on six HuggingFace datasets show BG-LMC delivers compression near the entropy limit with substantially faster encode times than traditional encoders, and PLMC scales to multi-core setups with throughput surpassing parallel gzip and BZ2. The findings demonstrate practical impact by enabling higher-frequency checkpoints with reduced CPU and storage overhead, improving resilience to failures in large-scale training.

Abstract

During the training of Large Language Models (LLMs), tensor data is periodically "checkpointed" to persistent storage to allow recovery of work done in the event of failure. The volume of data that must be copied during each checkpoint, even when using reduced-precision representations such as bfloat16, often reaches hundreds of gigabytes. Furthermore, the data must be moved across a network and written to a storage system before the next epoch occurs. With a view to ultimately building an optimized checkpointing solution, this paper presents experimental analysis of checkpoint data used to derive a design that maximizes the use of lossless compression to reduce the volume of data. We examine how tensor data and its compressibility evolve during model training and evaluate the efficacy of existing common off-the-shelf general purpose compression engines combined with known data optimization techniques such as byte-grouping and incremental delta compression. Leveraging our analysis we have built an effective compression solution, known as Language Model Compressor (LMC), which is based on byte-grouping and Huffman encoding. LMC offers more compression performance than the best alternative (BZ2) but with an order-of-magnitude reduction in the time needed to perform the compression. We show that a 16-core parallel implementation of LMC can attain compression and decompression throughput of 2.78 GiB/s and 3.76 GiB/s respectively. This increase in performance ultimately reduces the CPU resources needed and provides more time to copy the data to the storage system before the next epoch thus allowing for higher-frequency checkpoints.

Lossless Compression for LLM Tensor Incremental Snapshots

TL;DR

The paper tackles the large data burden of checkpointing during LLM training by analyzing the structure and evolution of tensor deltas and proposing a lossless compression pipeline. It introduces Byte-Grouping with Run-Length Encoding and a block-adaptive Huffman coder (LMC), extended to a parallel variant (PLMC) to achieve high throughput. Empirical results on six HuggingFace datasets show BG-LMC delivers compression near the entropy limit with substantially faster encode times than traditional encoders, and PLMC scales to multi-core setups with throughput surpassing parallel gzip and BZ2. The findings demonstrate practical impact by enabling higher-frequency checkpoints with reduced CPU and storage overhead, improving resilience to failures in large-scale training.

Abstract

During the training of Large Language Models (LLMs), tensor data is periodically "checkpointed" to persistent storage to allow recovery of work done in the event of failure. The volume of data that must be copied during each checkpoint, even when using reduced-precision representations such as bfloat16, often reaches hundreds of gigabytes. Furthermore, the data must be moved across a network and written to a storage system before the next epoch occurs. With a view to ultimately building an optimized checkpointing solution, this paper presents experimental analysis of checkpoint data used to derive a design that maximizes the use of lossless compression to reduce the volume of data. We examine how tensor data and its compressibility evolve during model training and evaluate the efficacy of existing common off-the-shelf general purpose compression engines combined with known data optimization techniques such as byte-grouping and incremental delta compression. Leveraging our analysis we have built an effective compression solution, known as Language Model Compressor (LMC), which is based on byte-grouping and Huffman encoding. LMC offers more compression performance than the best alternative (BZ2) but with an order-of-magnitude reduction in the time needed to perform the compression. We show that a 16-core parallel implementation of LMC can attain compression and decompression throughput of 2.78 GiB/s and 3.76 GiB/s respectively. This increase in performance ultimately reduces the CPU resources needed and provides more time to copy the data to the storage system before the next epoch thus allowing for higher-frequency checkpoints.
Paper Structure (17 sections, 4 equations, 14 figures, 2 tables)

This paper contains 17 sections, 4 equations, 14 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background: The Nature of LLM Tensors
  3. Floating-Point Representations
  4. Analysis of LLM Checkpoint Data
  5. Converging Behavior
  6. Proposed Compression Algorithm
  7. Byte-Grouping and RLE
  8. Estimating compressibility with Entropy
  9. Large Model Compression (LMC)
  10. Parallel LMC (PLMC)
  11. Experimental Results
  12. Single-threaded Performance
  13. Compression Ratios and Processing Time
  14. Compression Variance Over Time
  15. Multi-core Scale-up Performance
  16. ...and 2 more sections

Figures (14)

  • Figure 1: Tensor Data Shards Evolving
  • Figure 2: Functional description of a biological neuron’s structure fundamentals2017
  • Figure 3: LLM Floating Point Formats
  • Figure 4: Sample Self-Attention Weights over Time from BLOOM Data Set
  • Figure 5: Bit Distribution for BLOOM Self-Attention QKV Weights Shard 0
  • ...and 9 more figures