Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Model-Driven Lossless Compression Algorithm Resistant to Mismatch

Cordelia Hu, Jennifer Tang

TL;DR

This work tackles the problem of non-determinism in model-driven, lossless compression by introducing a mismatch-tolerant encoding scheme that does not rely on arithmetic coding. The proposed algorithm certifies robustness under a structured mismatch via a bound $| ln p(x) - \nln p'(x)| < \nln c$, and communicates tokens through a bucketed interval scheme with a short unique prefix, ensuring correct decoding under the bound. The authors provide a correctness proof, derive performance bounds under a power-law model for token distributions with $\alpha \approx 1.8$, and conduct experiments on real text data showing compression ratios well above gzip for several settings, while maintaining high decoding accuracy up to moderate mismatch levels. This demonstrates practical, mismatch-resilient compression that can be deployed with learned predictors, offering meaningful gains in robustness and efficiency for model-driven compression pipelines.

Abstract

Due to the fundamental connection between next-symbol prediction and compression, modern predictive models, such as large language models (LLMs), can be combined with entropy coding to achieve compression rates that surpass those of standard compression algorithms. However, this approach relies on the assumption that the predictive model produces identical output distributions at both the encoder and decoder, since even small mismatches can cause the decoding to fail. This assumption often fails with complex predictive models, particularly those based on neural networks, a phenomenon referred to as non-determinism. In this work, we propose a new compression algorithm based on next-token prediction that is robust to arbitrarily large, but structured, prediction mismatches. We prove the correctness of the proposed scheme under a formal mismatch certification, characterize its theoretical performance, and validate it experimentally on real datasets. Our results demonstrate reliable operation within the certified mismatch regime while achieving compression ratios that exceed those of commonly used compression methods.

A Model-Driven Lossless Compression Algorithm Resistant to Mismatch

TL;DR

This work tackles the problem of non-determinism in model-driven, lossless compression by introducing a mismatch-tolerant encoding scheme that does not rely on arithmetic coding. The proposed algorithm certifies robustness under a structured mismatch via a bound , and communicates tokens through a bucketed interval scheme with a short unique prefix, ensuring correct decoding under the bound. The authors provide a correctness proof, derive performance bounds under a power-law model for token distributions with , and conduct experiments on real text data showing compression ratios well above gzip for several settings, while maintaining high decoding accuracy up to moderate mismatch levels. This demonstrates practical, mismatch-resilient compression that can be deployed with learned predictors, offering meaningful gains in robustness and efficiency for model-driven compression pipelines.

Abstract

Due to the fundamental connection between next-symbol prediction and compression, modern predictive models, such as large language models (LLMs), can be combined with entropy coding to achieve compression rates that surpass those of standard compression algorithms. However, this approach relies on the assumption that the predictive model produces identical output distributions at both the encoder and decoder, since even small mismatches can cause the decoding to fail. This assumption often fails with complex predictive models, particularly those based on neural networks, a phenomenon referred to as non-determinism. In this work, we propose a new compression algorithm based on next-token prediction that is robust to arbitrarily large, but structured, prediction mismatches. We prove the correctness of the proposed scheme under a formal mismatch certification, characterize its theoretical performance, and validate it experimentally on real datasets. Our results demonstrate reliable operation within the certified mismatch regime while achieving compression ratios that exceed those of commonly used compression methods.
Paper Structure (25 sections, 4 theorems, 36 equations, 5 figures)

This paper contains 25 sections, 4 theorems, 36 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Model-driven Compression
  3. Non-Determinism and Prediction Mismatch
  4. Previous Work
  5. Main Contributions
  6. Notation
  7. Problem Setting and Mismatch Assumptions
  8. Algorithm
  9. Encoder
  10. Encoding Subprocess
  11. Decoder and correctness proof
  12. Decoder Subprocess
  13. Performance Analysis
  14. Interval Selection Analysis
  15. Experimental Verification and Measurement of the Power Law
  16. ...and 10 more sections

Key Result

Proposition 1

Given Assumption eq::assumption_bounded and the correctness of all prior steps in the decoding,

Figures (5)

  • Figure 1: Fifty probability distributions produced by Meta Llama 3.1, overlayed onto each other; snippets of Hamlet were provided as context.
  • Figure 2: Regression slopes of over $100,000$ LLM probability distributions, with context from random Wikipedia articles. Slopes are estimated from linear regression after logarithmically transforming both rank and probability.
  • Figure 3: Compression ratios achieved in our experiements. Error bars shows the standard deviation in compression ratios. The dashed horizontal lines represent the mean compression ratios of the commonly used gzip algorithm.
  • Figure 4: Decompression accuracies of Wikipedia articles, sorted by $q$ value.
  • Figure 5: Histogram of the number of bits used to encode each token ($|y_i|$) in Wikipedia article tests, given $q = 0.3$.

Theorems & Definitions (7)

  • Proposition 1
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Proposition 2
  • proof