Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Synchronizing Probabilities in Model-Driven Lossless Compression

Aviv Adler, Jennifer Tang

TL;DR

This work tackles prediction mismatch in model-driven lossless compression caused by LLM non-determinism, formalizing a bounded-logit deviation setting and its impact on decoding. It introduces PMATIC, a model-agnostic Probability Matching Interval Coding scheme that uses bin-based quantization of next-bit probabilities and helper bits to ensure encoder/decoder agreement, guaranteeing correct decoding when the mismatch is bounded by $\varepsilon$ (with $\delta$ linked to $\varepsilon$). The authors provide a correctness proof and derive compression-loss bounds, showing the total overhead scales as $O(\sqrt{\delta \log(1/\delta)})$ with an optimally chosen bin width $r$. Empirically, PMATIC on text with Llama 3.1 demonstrates robustness to synthetic prediction mismatch and achieves substantially better compression than gzip, illustrating practical viability for robust, high-performance model-driven compression.

Abstract

It is well-known in the field of lossless data compression that probabilistic next-symbol prediction can be used to compress sequences of symbols. Deep neural networks are able to capture rich dependencies in data, offering a powerful means of estimating these probabilities and hence an avenue towards more effective compression algorithms. However, both compressor and decompressor must have exactly matching predictions; even small non-deterministic differences (which often happen with learned models due to hardware, software, or computation order) can lead to cascading decoding failures. In this paper, we formalize the problem of prediction mismatch in model-driven compression, and introduce Probability Matching Interval Coding (PMATIC), a model-agnostic algorithm that tolerates bounded prediction mismatch with low overhead. PMATIC works with the predicted probabilities, making it compatible as a drop-in replacement for the arithmetic encoder in model-driven compression tools. We show theoretical correctness and performance bounds for PMATIC, and validate these results on text data. These results confirm that, when paired an advanced prediction model, PMATIC is robust to prediction mismatch while achieving compression rates that out-perform standard modern compression tools.

Synchronizing Probabilities in Model-Driven Lossless Compression

TL;DR

This work tackles prediction mismatch in model-driven lossless compression caused by LLM non-determinism, formalizing a bounded-logit deviation setting and its impact on decoding. It introduces PMATIC, a model-agnostic Probability Matching Interval Coding scheme that uses bin-based quantization of next-bit probabilities and helper bits to ensure encoder/decoder agreement, guaranteeing correct decoding when the mismatch is bounded by (with linked to ). The authors provide a correctness proof and derive compression-loss bounds, showing the total overhead scales as with an optimally chosen bin width . Empirically, PMATIC on text with Llama 3.1 demonstrates robustness to synthetic prediction mismatch and achieves substantially better compression than gzip, illustrating practical viability for robust, high-performance model-driven compression.

Abstract

It is well-known in the field of lossless data compression that probabilistic next-symbol prediction can be used to compress sequences of symbols. Deep neural networks are able to capture rich dependencies in data, offering a powerful means of estimating these probabilities and hence an avenue towards more effective compression algorithms. However, both compressor and decompressor must have exactly matching predictions; even small non-deterministic differences (which often happen with learned models due to hardware, software, or computation order) can lead to cascading decoding failures. In this paper, we formalize the problem of prediction mismatch in model-driven compression, and introduce Probability Matching Interval Coding (PMATIC), a model-agnostic algorithm that tolerates bounded prediction mismatch with low overhead. PMATIC works with the predicted probabilities, making it compatible as a drop-in replacement for the arithmetic encoder in model-driven compression tools. We show theoretical correctness and performance bounds for PMATIC, and validate these results on text data. These results confirm that, when paired an advanced prediction model, PMATIC is robust to prediction mismatch while achieving compression rates that out-perform standard modern compression tools.
Paper Structure (17 sections, 2 theorems, 15 equations, 1 figure)

This paper contains 17 sections, 2 theorems, 15 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Model-Driven Lossless Compression
  3. LLM Non-Determinism and Prediction Mismatch
  4. Contributions
  5. Problem Statement
  6. The Bounded Prediction Mismatch Setting
  7. The PMATIC Algorithm
  8. The PMATIC Encoder
  9. The PMATIC Decoder
  10. Analysis
  11. Correctness
  12. Compression Loss
  13. Experiments
  14. Setup
  15. Results
  16. ...and 2 more sections

Key Result

Proposition 1

If ${\boldsymbol{u}}, {\boldsymbol{v}}$ induce probability distributions ${\boldsymbol{p}} = \mathop{\mathrm{\mathrm{softmax}}}\nolimits({\boldsymbol{u}})$ and ${\boldsymbol{q}} = \mathop{\mathrm{\mathrm{softmax}}}\nolimits({\boldsymbol{v}})$ over $\mathop{\mathrm{{\mathcal{A}}}}\nolimits$, and $\|

Figures (1)

  • Figure 1: Compression results of PMATIC algorithm with two robustness levels compared to 1) when the same LLM-driven compression is used when there is no mismatch and 2) gzip (modern standard for text compression). The total raw size of the 1000 random Wikipedia articles is $2,992,016$ bytes. We use the first $\approx 10$ MB of enwik8, with total raw size $9,923,563$ bytes.

Theorems & Definitions (5)

  • Definition 1
  • Proposition 1
  • proof
  • Theorem 1
  • proof