Data Compression with Stochastic Codes

TL;DR

This work surveys relative entropy coding (REC), a stochastic-code framework that extends lossy data compression beyond fixed quantisers by allowing arbitrary reconstruction distributions $P_{at X|X}$ and leveraging shared randomness. It organizes a spectrum of constructions—from rejection sampling and Poisson functional representations to dithered quantisers and selection samplers—while analysing runtime bounds and practical constraints, including the necessity of synchronization. The paper demonstrates REC's potential through applications in learned compression, realism-aware coding, privacy-preserving mechanisms, and communication-efficient reinforcement learning, and discusses extensions to multidimensional data, channel-code-based REC, and advanced sampling techniques. It also candidly discusses limitations, notably speed and synchronization challenges, and positions REC as a promising direction for end-to-end optimization and rate-distortion-realism trade-offs, pending advances in fast, robust implementations.

Abstract

Machine learning has had a major impact on data compression over the last decade and inspired many new, exciting theoretical and applied questions. This paper describes one such direction -- relative entropy coding -- which focuses on constructing stochastic codes, primarily as an alternative to quantisation and entropy coding in lossy source coding. Our primary aim is to provide a broad overview of the topic, with an emphasis on the computational and practical aspects currently missing from the literature. Our goal is threefold: for the curious reader, we aim to provide an intuitive picture of the field and convince them that relative entropy coding is a simple yet exciting emerging field in data compression research. For a reader interested in applied research on lossy data compression, we provide an account of the most salient contemporary applications. Finally, for the reader who has heard of relative entropy coding but has never been quite sure what it is or how the algorithms fit together, we hope to illustrate how simple and elegant the underlying constructions are.

Figures (4)

• Figure 1: An illustration of a "reverse Cardan" grille: the cover text is fixed ahead of time, and the grille is constructed one hole at a time to reveal the next letter of the desired message. Left: The first two paragraphs of "The hitchhiker's guide to the galaxy" by Douglas Adams adams2009hitchhiker. Right: A grille applied to the same two paragraphs, revealing an important fact about the meaning of life, the universe, and everything.
• Figure 2: An illustration of rejection sampling with target $Q = \mathcal{N}(1, 1/4)$ and $P = \mathcal{N}(0, 1)$. Rejection sampling simulates points $(Y_k, U_k)$ with $Y_k \sim P$ and $U_k \sim \mathrm{Unif}(0, 1)$. If the simulated point falls within the blue-shaded region, the algorithm terminates. The illustration also demonstrates the myopic nature of rejection sampling: two consecutive samples, $Y_k$ and $Y_{k + 1}$, might fall in the same small set $A$, but only the latter sample $Y_{k + 1}$ is accepted.
• Figure 3: Illustration of relative entropy coding for the pair of random variables $X, Y \sim P_{X, Y}$ using a selection sampler. The sender Anna and the receiver Béla share a sequence of i.i.d. $P_Y$-distributed samples as their common randomness $Z$. Then, upon receiving a source sample $X \sim P_{X}$, A uses a selection rule $N$ that selects one of the samples in the shared sequence such that $Y_K \sim P_{Y \mid X}$. Since the selected index $K$ is discrete, A uses an appropriate entropy coding algorithm to efficiently encode $N$ and transmit it to B. Finally, B can recover a $P_{Y \mid X}$-distributed sample by decoding $N$ and selecting the $N$th sample in the shared sequence.
• Figure 4: Remote reinforcement learning: a controller (with reward access) guides a remote actor over a rate-constrained channel.