Float8@2bits: Entropy Coding Enables Data-Free Model Compression

TL;DR

EntQuant addresses the memory bottleneck of deploying large language models by decoupling storage cost from numerical precision through entropy coding. The method quantizes weights in a high-precision base (Float8/Int8) and uses GPU-accelerated asymmetric numeral systems (ANS) to losslessly compress to effective rates as low as $2$ bits per parameter, with on-device decoding during inference. It achieves state-of-the-art results among data-free methods and competitive performance against calibration- and fine-tuning-based quantization, including instruction-tuned models, while maintaining acceptable inference speed. This data-free, entropy-driven approach enables practical deployment of much larger models under tight memory constraints, reducing the need for access to training data or expensive retraining.

Abstract

Post-training compression is currently divided into two contrasting regimes. On the one hand, fast, data-free, and model-agnostic methods (e.g., NF4 or HQQ) offer maximum accessibility but suffer from functional collapse at extreme bit-rates below 4 bits. On the other hand, techniques leveraging calibration data or extensive recovery training achieve superior fidelity but impose high computational constraints and face uncertain robustness under data distribution shifts. We introduce EntQuant, the first framework to unite the advantages of these distinct paradigms. By matching the performance of data-dependent methods with the speed and universality of data-free techniques, EntQuant enables practical utility in the extreme compression regime. Our method decouples numerical precision from storage cost via entropy coding, compressing a 70B parameter model in less than 30 minutes. We demonstrate that EntQuant does not only achieve state-of-the-art results on standard evaluation sets and models, but also retains functional performance on more complex benchmarks with instruction-tuned models, all at modest inference overhead.

Figures (12)

• Figure 1: EntQuant compresses instruction-tuned models without data, performing well on several advanced benchmarks. Numbers above the size axis indicate effective bits per parameter.
• Figure 2: Illustration of 4-bit weight encoding with EntQuant, compared to fixed bit-width quantization. Boxes illustrate weight matrices at different representations with weight histograms above the weight matrix. Note that the number of colors and histogram bins is reduced for illustrative purposes. Weights optimized with EntQuant have more diverse parameters compared to fixed bit-width representations. With entropy coding, more common parameter values can be stored efficiently, see \ref{['tab:unique_val']}. \ref{['fig:inference_pipeline']} depicts inference with EntQuant.
• Figure 3: Visualization of EntQuant's inference pipeline.
• Figure 4: Memory-perplexity trade-off on C4 for LLaMA-2 7B, 13B, 70B. EntQuant spans a smooth Pareto front enabling fine-grained compression-performance trade-offs. Surface areas of dots are proportional to the bit-rate of each model. Float8 is entropy-encoded as well, leading to approximately $6.5$ bits per parameter.
• Figure 5: Inference throughput for LLaMA-2 13B in a standard prefill-decoding setting (input context length 512 tokens and 256 tokens generated). See \ref{['fig:efficiency_full_throughput', 'fig:efficiency_full_latency', 'fig:efficiency_full_peak_memory']} for more results.