Lossless Compression of Large Language Model-Generated Text via Next-Token Prediction

TL;DR

This work addresses the escalating volume of LLM-generated text and the need for lossless compression tailored to such data. It introduces an LLM-based compression framework that leverages next-token prediction and entropy coding, formalized through context-based probability models and arithmetic coding. Across 14 LLMs and 8 diverse datasets, the approach consistently achieves >20× compression, far surpassing traditional baselines and showing robustness to model size, dataset type, and scale. The results imply a practical path for efficient storage and management of AI-generated text in real-world systems, with insights into chunk-size and domain-tuning effects that guide deployment choices.

Abstract

As large language models (LLMs) continue to be deployed and utilized across domains, the volume of LLM-generated data is growing rapidly. This trend highlights the increasing importance of effective and lossless compression for such data in modern text management systems. However, compressing LLM-generated data presents unique challenges compared to traditional human- or machine-generated content. Traditional machine-generated data is typically derived from computational processes or device outputs, often highly structured and limited to low-level elements like labels or numerical values. This structure enables conventional lossless compressors to perform efficiently. In contrast, LLM-generated data is more complex and diverse, requiring new approaches for effective compression. In this work, we conduct the first systematic investigation of lossless compression techniques tailored specifically to LLM-generated data. Notably, because LLMs are trained via next-token prediction, we find that LLM-generated data is highly predictable for the models themselves. This predictability enables LLMs to serve as efficient compressors of their own outputs. Through extensive experiments with 14 representative LLMs and 8 LLM-generated datasets from diverse domains, we show that LLM-based prediction methods achieve remarkable compression rates, exceeding 20x, far surpassing the 3x rate achieved by Gzip, a widely used general-purpose compressor. Furthermore, this advantage holds across different LLM sizes and dataset types, demonstrating the robustness and practicality of LLM-based methods in lossless text compression under generative AI workloads.

Figures (9)

• Figure 1: Trend of increasing LLM-generated data.
• Figure 2: N-gram frequency count for LLM-generated synthetic data.
• Figure 3: LLM-generated data distribution.
• Figure 4: General process of Neural-based Compression.
• Figure 5: Compression ratio comparison across different Llama models. Larger models achieve higher compression ratios, demonstrating the advantage of increased model capacity in leveraging contextual information. Instruction-tuned variants generally perform slightly worse than their base counterparts on most datasets but achieve better compression on question-answering datasets.