Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I)

Nnamdi Aghanya, Jun Li, Kewei Wang

TL;DR

The study addresses lossy text compression with LLMs by proposing Error-Bounded Predictive Coding (EPC), which uses a Masked Language Model as a decompressor and transmits only rank-based corrections when top predictions fail, providing a controllable residual channel. EPC is evaluated against Predictive Masking (PM) and a transform-based VQ+RE baseline on WikiText-103, demonstrating substantial rate–distortion gains; for example, reaching CharFid around 0.98 requires 52–63% fewer bits with EPC than PM, and RoBERTa-base achieves near-lossless fidelity at roughly $1.1$ BPC. The methodology includes a three-stage pipeline (model specialisation, compression codecs, reconstruction/refinement), exact entropy coding with AC/rANS, and a robust evaluation framework across multiple MLMs and masking/threshold parameters. These findings suggest EPC offers a practical, tunable approach to high-fidelity lossy text compression, with clear directions for improvements such as joint training, adaptive budgets, and streaming implementations.

Abstract

Large Language Models (LLMs) can achieve near-optimal lossless compression by acting as powerful probability models. We investigate their use in the lossy domain, where reconstruction fidelity is traded for higher compression ratios. This paper introduces Error-Bounded Predictive Coding (EPC), a lossy text codec that leverages a Masked Language Model (MLM) as a decompressor. Instead of storing a subset of original tokens, EPC allows the model to predict masked content and stores minimal, rank-based corrections only when the model's top prediction is incorrect. This creates a residual channel that offers continuous rate-distortion control. We compare EPC to a simpler Predictive Masking (PM) baseline and a transform-based Vector Quantisation with a Residual Patch (VQ+RE) approach. Through an evaluation that includes precise bit accounting and rate-distortion analysis, we demonstrate that EPC consistently dominates PM, offering superior fidelity at a significantly lower bit rate by more efficiently utilising the model's intrinsic knowledge.

The Lossy Horizon: Error-Bounded Predictive Coding for Lossy Text Compression (Episode I)

TL;DR

The study addresses lossy text compression with LLMs by proposing Error-Bounded Predictive Coding (EPC), which uses a Masked Language Model as a decompressor and transmits only rank-based corrections when top predictions fail, providing a controllable residual channel. EPC is evaluated against Predictive Masking (PM) and a transform-based VQ+RE baseline on WikiText-103, demonstrating substantial rate–distortion gains; for example, reaching CharFid around 0.98 requires 52–63% fewer bits with EPC than PM, and RoBERTa-base achieves near-lossless fidelity at roughly BPC. The methodology includes a three-stage pipeline (model specialisation, compression codecs, reconstruction/refinement), exact entropy coding with AC/rANS, and a robust evaluation framework across multiple MLMs and masking/threshold parameters. These findings suggest EPC offers a practical, tunable approach to high-fidelity lossy text compression, with clear directions for improvements such as joint training, adaptive budgets, and streaming implementations.

Abstract

Large Language Models (LLMs) can achieve near-optimal lossless compression by acting as powerful probability models. We investigate their use in the lossy domain, where reconstruction fidelity is traded for higher compression ratios. This paper introduces Error-Bounded Predictive Coding (EPC), a lossy text codec that leverages a Masked Language Model (MLM) as a decompressor. Instead of storing a subset of original tokens, EPC allows the model to predict masked content and stores minimal, rank-based corrections only when the model's top prediction is incorrect. This creates a residual channel that offers continuous rate-distortion control. We compare EPC to a simpler Predictive Masking (PM) baseline and a transform-based Vector Quantisation with a Residual Patch (VQ+RE) approach. Through an evaluation that includes precise bit accounting and rate-distortion analysis, we demonstrate that EPC consistently dominates PM, offering superior fidelity at a significantly lower bit rate by more efficiently utilising the model's intrinsic knowledge.
Paper Structure (37 sections, 10 equations, 3 figures, 2 tables)

This paper contains 37 sections, 10 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Models and Datasets
  5. Metrics and Cost Accounting
  6. Entropy coding details (AC/rANS).
  7. Compression Pipeline
  8. Stage 1: Model Specialisation.
  9. Stage 2: Compression Codecs.
  10. Stage 3: Reconstruction and Refinement.
  11. Implementation and Evaluation Protocol
  12. Experiments and Results
  13. Protocol.
  14. Main rate–distortion results (CharFid)
  15. Behaviour across masking rates.
  16. ...and 22 more sections

Figures (3)

  • Figure 4: Rate–distortion (CharFid vs. BPC). EPC consistently Pareto-dominates PM in the high-fidelity region. VQ+RE occupies a different RD regime, trading much lower rates for substantially lower fidelity. Error bars (where visible) show $\pm1$ sd over seeds; PM/EPC variances are negligible.
  • Figure 5: ChrF/BERTScore vs. BPC. PM degrades as $p_{\text{mask}}$ grows; EPC stays saturated across rates, especially with RoBERTa.
  • Figure 6: EPC across models. RoBERTa delivers the strongest RD curve; BERT-base and DistilBERT are slightly behind but still markedly outperform PM.