Revisiting Data Compression with Language Modeling

TL;DR

This work examines the feasibility and practical effectiveness of using large language models for lossless data compression via probabilistic modeling and arithmetic coding. It demonstrates a state-of-the-art adjusted compression rate of about $0.18$ on enwik9 without additional training and extends evaluation to non-English data and byte-stream inputs. The authors investigate context-length extension and post-training quantization (GPTQ and HQQ) to reduce model size while preserving compression performance, finding that substantial weight-precision reductions can maintain competitive $\gamma_a$. They also compare multiple byte-stream tokenization strategies, revealing that treating bytes as integers or mapped tokens yields the strongest compression, with text-oriented approaches performing comparatively poorer in adjusted metrics. Overall, the study highlights the potential and limitations of LLM-based compression across text, code, multilingual data, and raw byte streams, pointing to directions for practical, scalable data compressors specialized for compression tasks.

Abstract

In this report, we investigate the potential use of large language models (LLM's) in the task of data compression. Previous works have demonstrated promising results in applying LLM's towards compressing not only text, but also a wide range of multi-modal data. Despite the favorable performance achieved, there still remains several practical questions that pose a challenge towards replacing existing data compression algorithms with LLM's. In this work, we explore different methods to achieve a lower adjusted compression rate using LLM's as data compressors. In comparison to previous works, we were able to achieve a new state-of-the-art (SOTA) adjusted compression rate of around $18\%$ on the enwik9 dataset without additional model training. Furthermore, we explore the use of LLM's in compressing non-English data, code data, byte stream sequences. We show that while LLM's excel in compressing data in text-dominant domains, their ability in compressing non-natural text sequences still remain competitive if configured in the right way.

Figures (8)

• Figure 1: Arithmetic coding in action. (figure from deletang2024language) In this example, there are 3 possible symbols: {X,I,A}. When encoding a symbol, the probability distribution of all symbols is generated by a probabilistic model (e.g., an LLM). The range for which a valid encoding narrows as the number of encoded symbols increase. The final encoded symbol (e.g., $0101010$) must lie in an interval that represents the original sequence of symbols (e.g., AIXI).
• Figure 2: Byte Mapping for Llama 3 and Qwen Model. On the left is mapped byte tokens for the Llama 3 Touvron2023Llama2O model. On the right is the mapped byte tokens for the Qwen qwen model. Notice how the two mappings are seemingly identical, except that their mapped tokens are shifted by approximately $200$. This is most likely due to the fact that both tokenizers are based on Byte-Pair Encoding. The different in token value shift is due to different configurations of the two tokenizers.
• Figure 3: Context Length and Compression Rate. Shown above is the raw compression rate $\gamma_r$ as a function of the context length of an LLM. Notice the steady decrease in $\gamma_r$ as the context length increases. Since Llama-3.2-1B and Qwen2-0.5B both naively support context lengths of up to $128$k tokens, their compression ration is expected to improve for even longer context lengths. Since SmolLM models only support up to $2048$ tokens, $\gamma_r$ increases when we exceed this limit.
• Figure 4: Raw Compression Rate Under Different Quantization Schemes. Show above is the raw compression rate $\gamma_r$ under HQQ (left) and GPTQ (right) quantization. The $16$-bit representation is the original un-quantized representation. Notice how HQQ, a calibration-free quantization algorithm, retains better predictive performance compared to GPTQ.
• Figure 5: Adjusted Compression Rate Under Different Quantization Schemes. Show above is the adjusted compression rate $\gamma_a$ under HQQ (left) and GPTQ (right) quantization. The $16$-bit representation is the original un-quantized representation. Compression rates exceeding $1.0$ have been labeled and capped. In comparison to the baseline general purpose compressor, LLM's under the $0.5$B parameter range have all exceeded Gzip's compression rate with $3$-bit compression.