70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)

TL;DR

This work tackles the memory bottlenecks of large BF16 foundation models by introducing Dynamic-Length Float (DFloat11), a lossless compression that encodes BF16 exponents with Huffman coding while preserving sign and mantissa. The approach achieves approximately 70% model-size reduction with bit-for-bit identical outputs, facilitated by a GPU-optimized decompression kernel using hierarchical LUTs and transformer-block-level batching. Empirical results across Llama 3, Qwen 3, Mistral 3, and diffusion transformers show no accuracy loss and substantial throughput and memory benefits, enabling longer generation and even single-node deployment of extremely large models. DF11 thus provides a practical, lossless alternative to lossy quantization for efficient GPU inference on large-scale LLMs and diffusion models.

Abstract

Large-scale AI models, such as Large Language Models (LLMs) and Diffusion Models (DMs), have grown rapidly in size, creating significant challenges for efficient deployment on resource-constrained hardware. In this paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression framework that reduces LLM and DM size by 30% while preserving outputs that are bit-for-bit identical to the original model. DFloat11 is motivated by the low entropy in the BFloat16 weight representation of LLMs, which reveals significant inefficiency in the existing storage format. By applying entropy coding, DFloat11 assigns dynamic-length encodings to weights based on frequency, achieving near information-optimal compression without any loss of precision. To facilitate efficient inference with dynamic-length encodings, we develop a custom GPU kernel for fast online decompression. Our design incorporates the following: (i) compact, hierarchical lookup tables (LUTs) that fit within GPU SRAM for efficient decoding, (ii) a two-phase GPU kernel for coordinating thread read/write positions using lightweight auxiliary variables, and (iii) transformer-block-level decompression to minimize latency. Experiments on Llama 3.3, Qwen 3, Mistral 3, FLUX.1, and others validate our hypothesis that DFloat11 achieves around 30% model size reduction while preserving bit-for-bit identical outputs. Compared to a potential alternative of offloading parts of an uncompressed model to the CPU to meet memory constraints, DFloat11 achieves 2.3--46.2x higher throughput in token generation. With a fixed GPU memory budget, DFloat11 enables 5.7--14.9x longer generation lengths than uncompressed models. Notably, our method enables lossless inference of Llama 3.1 405B, an 810GB model, on a single node equipped with 8x80GB GPUs.

Figures (13)

• Figure 1: (Left) The allocation of bits for the components of BFloat16. (Right 3) The Shannon entropy of the components (sign, exponent, mantissa) of BFloat16 weights in various LLMs.
• Figure 2: Our proposed format Dynamic-Length Float for compressing BFloat16 weights of LLMs losslessly down to 11 bits. The exponents are compressed via Huffman coding, while the sign and mantissa bits remain uncompressed.
• Figure 3: (Left) The Huffman tree is decomposed into a set of non-overlapping subtrees, each corresponding to a compact lookup table (LUT). These hierarchical LUTs reside in GPU SRAM to enable efficient Huffman decoding via array lookups. (Right) Each thread decodes $n$ bytes of encoded exponents. The array Gaps stores the bit offset of the first element assigned to each thread, while the array Block Output Positions stores the index of the first element for each thread block.
• Figure 4: Comparison of throughput (top row) and latency (bottom row) for token decoding using the original BF16 models and their DF11-compressed counterparts. Portions of the BF16 models are offloaded to the CPU due to GPU memory constraints.
• Figure 5: Comparison of GPU memory consumption between BF16 models and DF11 counterparts. The DF11 models support 5.70--14.86$\times$ longer context lengths by allowing more GPU memory to be used for storing the KV cache. "O.O.M." means out of memory.