Test-Time Steering for Lossless Text Compression via Weighted Product of Experts

TL;DR

The paper tackles lossless text compression under distribution shift by introducing test-time steering via Weighted Product of Experts (wPoE), which blends a training-free universal compressor with a pretrained autoregressive language model. It provides a theoretical guarantee that the wPoE can match or outperform the best individual expert and demonstrates empirically that this approach improves compression on diverse datasets without any fine-tuning. The method efficiently learns a single optimal weight from minimal data and extends naturally to multiple experts, offering practical gains in compute and memory usage. Overall, wPoE offers a robust, scalable way to generalize neural text compression to unseen distributions while preserving the simplicity of training-free adaptation.

Abstract

Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively. Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE). At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model. Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.

Figures (3)

• Figure 1: A diagram of our compression pipeline: we apply a weighted-product-of-experts approach to steer the probability distribution during inference. After obtaining the steered distribution, we use arithmetic coding to compress the sequence. As shown in the figure, the sequence ‘I, trust, in, thee’ can be compressed as any real number in the interval [0.3728, 0.3820), as introduced in Section \ref{['subsect:ac_autoregressive_model']}.
• Figure 2: Figure (a) illustrates the compression rate on the OOD dataset code for three Our wPoE models of different sizes, under varying values of $\alpha$. Figure (b) shows $\alpha$ changes as the number of iterations increases when using L-BFGS optimizer. The annotated circular points represent the optimal $\alpha$ found through grid search. The points marked with asterisks indicate the converged $\alpha$ values after optimization.
• Figure 3: We compare our method against fine-tuning on byte-level decoder-only Transformers, given a sequence containing 2048 bytes of out-of-distribution data. Our method is more robust to overfitting. Consistent with Figure \ref{['fig:alpha_optimize_vs_gridsearch']} (b), we initialize $\alpha$ to be 0.0.

Theorems & Definitions (5)

• Proposition 1
• Lemma 1
• Lemma 2
• proof : Proof of Lemma \ref{['lemma:proof_inequality_2distribution']}
• proof : Proof of Lemma \ref{['lemma:proof_inequality']}