Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression

Fazal Mittu, Yihuan Bu, Akshat Gupta, Ashok Devireddy, Alp Eren Ozdarendeli, Anant Singh, Gopala Anumanchipalli

TL;DR

This work presents FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely, and takes the first step towards making lossless text compression with LLMs a reality.

Abstract

While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50\%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.

FineZip : Pushing the Limits of Large Language Models for Practical Lossless Text Compression

TL;DR

This work presents FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely, and takes the first step towards making lossless text compression with LLMs a reality.

Abstract

While the language modeling objective has been shown to be deeply connected with compression, it is surprising that modern LLMs are not employed in practical text compression systems. In this paper, we provide an in-depth analysis of neural network and transformer-based compression techniques to answer this question. We compare traditional text compression systems with neural network and LLM-based text compression methods. Although LLM-based systems significantly outperform conventional compression methods, they are highly impractical. Specifically, LLMZip, a recent text compression system using Llama3-8B requires 9.5 days to compress just 10 MB of text, although with huge improvements in compression ratios. To overcome this, we present FineZip - a novel LLM-based text compression system that combines ideas of online memorization and dynamic context to reduce the compression time immensely. FineZip can compress the above corpus in approximately 4 hours compared to 9.5 days, a 54 times improvement over LLMZip and comparable performance. FineZip outperforms traditional algorithmic compression methods with a large margin, improving compression ratios by approximately 50\%. With this work, we take the first step towards making lossless text compression with LLMs a reality. While FineZip presents a significant step in that direction, LLMs are still not a viable solution for large-scale text compression. We hope our work paves the way for future research and innovation to solve this problem.
Paper Structure (13 sections, 8 figures, 1 table)

This paper contains 13 sections, 8 figures, 1 table.

Table of Contents

  1. Introduction
  2. Introducing FineZip
  3. Experiments
  4. Modifications to LLMZip:
  5. FineZip Ablations
  6. Quantization:
  7. Conclusion
  8. Limitations
  9. Appendix
  10. Evaluating Traditional Compression Methods
  11. Double Compression Benchmark
  12. Context Size
  13. FineZip and Dataset Size

Figures (8)

  • Figure 1: System diagram for FineZip.
  • Figure 2: FineZip ablations for different fine-tune epochs
  • Figure 3: Compressing 10mb dataset with LLama-3 8B loaded with 4, 8, 16, and 32-bit precision. Purple bar shows compression ratio, red line shows time taken to compress. Each batch size was chosen to max out memory on a 48GB GPU.
  • Figure 4: Evaluating Baseline Compression Techniques Brotli, BZ2, and PPM on enwik8
  • Figure 5: Testing Traditional Compression Techniques Brotli, BZ2, and PPM on the ranks produced by compressing enwik8 with LLama2-7B finetuned for 64 epochs with r=16
  • ...and 3 more figures