Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Neural Distributed Compressor Discovers Binning

Ezgi Ozyilkan, Johannes Ballé, Elza Erkip

TL;DR

The paper tackles the one-shot Wyner–Ziv problem by proposing a data-driven, unstructured entropy-constrained vector quantization (ECVQ) framework that leverages decoder side information without requiring explicit source distributions. It introduces two neural formulations — a marginal variant with a classic entropy coder and a conditional variant paired with an ideal Slepian–Wolf coder — each backed by variational upper bounds on the entropy of the latent representation and its conditional entropy given the side information. Demonstrations on Gaussian and Laplacian sources reveal emergent binning in the source space and near–Wyner–Ziv performance, including optimal decoding within quantization regions and symmetry-driven binning in the Laplacian case. The work provides evidence that data-driven learning can recover fundamental WZ mechanisms like binning and joint index–side-information decoding, suggesting a practical path toward low-latency distributed compression without strong distributional priors.

Abstract

We consider lossy compression of an information source when the decoder has lossless access to a correlated one. This setup, also known as the Wyner-Ziv problem, is a special case of distributed source coding. To this day, practical approaches for the Wyner-Ziv problem have neither been fully developed nor heavily investigated. We propose a data-driven method based on machine learning that leverages the universal function approximation capability of artificial neural networks. We find that our neural network-based compression scheme, based on variational vector quantization, recovers some principles of the optimum theoretical solution of the Wyner-Ziv setup, such as binning in the source space as well as optimal combination of the quantization index and side information, for exemplary sources. These behaviors emerge although no structure exploiting knowledge of the source distributions was imposed. Binning is a widely used tool in information theoretic proofs and methods, and to our knowledge, this is the first time it has been explicitly observed to emerge from data-driven learning.

Neural Distributed Compressor Discovers Binning

TL;DR

The paper tackles the one-shot Wyner–Ziv problem by proposing a data-driven, unstructured entropy-constrained vector quantization (ECVQ) framework that leverages decoder side information without requiring explicit source distributions. It introduces two neural formulations — a marginal variant with a classic entropy coder and a conditional variant paired with an ideal Slepian–Wolf coder — each backed by variational upper bounds on the entropy of the latent representation and its conditional entropy given the side information. Demonstrations on Gaussian and Laplacian sources reveal emergent binning in the source space and near–Wyner–Ziv performance, including optimal decoding within quantization regions and symmetry-driven binning in the Laplacian case. The work provides evidence that data-driven learning can recover fundamental WZ mechanisms like binning and joint index–side-information decoding, suggesting a practical path toward low-latency distributed compression without strong distributional priors.

Abstract

We consider lossy compression of an information source when the decoder has lossless access to a correlated one. This setup, also known as the Wyner-Ziv problem, is a special case of distributed source coding. To this day, practical approaches for the Wyner-Ziv problem have neither been fully developed nor heavily investigated. We propose a data-driven method based on machine learning that leverages the universal function approximation capability of artificial neural networks. We find that our neural network-based compression scheme, based on variational vector quantization, recovers some principles of the optimum theoretical solution of the Wyner-Ziv setup, such as binning in the source space as well as optimal combination of the quantization index and side information, for exemplary sources. These behaviors emerge although no structure exploiting knowledge of the source distributions was imposed. Binning is a widely used tool in information theoretic proofs and methods, and to our knowledge, this is the first time it has been explicitly observed to emerge from data-driven learning.
Paper Structure (16 sections, 1 theorem, 20 equations, 9 figures)

This paper contains 16 sections, 1 theorem, 20 equations, 9 figures.

Table of Contents

  1. Introduction
  2. Contributions
  3. Popular ANN-Based Compressor Fails to Exploit Side Information
  4. Preliminaries, Setup and Prior Work
  5. Asymptotic Rate--Distortion Functions
  6. Operational Rate--Distortion Functions
  7. Our System Setup
  8. Relevant Lossy Compressor Designs in the Literature
  9. Neural upper bounds on Wyner--Ziv and Operational Schemes
  10. Estimating Neural Upper Bounds on Wyner--Ziv
  11. Induced Operational Schemes
  12. Experimental Setup
  13. Experimental Results and Discussion
  14. Gaussian Experiments
  15. Laplacian Experiments
  16. ...and 1 more sections

Key Result

Theorem 1

(Wyner--Ziv Theorem [1976]) Let $(X,Y)$ be correlated sources, drawn i.i.d. $\sim p(x,y)$, and let $d(x, \hat{x})$ be a single-letter distortion measure. The R-D function for $X$ with side information $Y$ available (noncausally) at the decoder is as follows: where the minimization operation is over all conditional probability distribution functions $p(u \vert x)$, and all functions $g(u,y)$ such

Figures (9)

  • Figure 1: One-shot lossy source coding with decoder-only side information.
  • Figure 2: Rate--distortion performances (R-D) obtained with Nonlinear Transform Coding (NTC) balle_journal, which is adapted to incorporate the available side information, that uses a variant of Eq. \ref{['eq:classic_rd']} as the objective function. Here, we consider a simple one-shot source coding with side information setting, where $X \sim \mathrm{Laplace}(0;1)$ and $Y= \mathrm{sgn}(X)$, i.e., the sign function of the input realization. We also plot the operational R-D function of the optimal entropy-constrained scalar quantizers (ECSQs), due to Sullivan sullivan, for the Laplacian and exponential sources. Note that the optimal ECSQ for the Laplacian source having sign information available at the decoder coincides with the optimal ECSQ for the exponential source.
  • Figure 3: Lossy source coding with decoder-only side information in the asymptotic blocklength regime.
  • Figure 4: The two lossy compression systems that we consider, based on upper bounds in Eqs. \ref{['eq:upper_bound_marg_2']} and \ref{['eq:upper_bound_cond_2']}: learned compressor using a classic entropy coder (a), and learned compressor using an ideal Slepian--Wolf coder (b). See Section \ref{['subsec:operational_schemes']} for the relevant discussion.
  • Figure 5: Visualizations (best viewed in color) of the learned encoder $u = \mathop{\mathrm{arg\,min}}\limits_{k} \ell_{\mathrm{m}}(k, x)$ (see Eq. \ref{['eq:f_m']}) and neural decoder $\hat{x} = g_{\boldsymbol{\phi}}(u,y)$ of the marginal formulation (see Eq. \ref{['eq:L_m']}), for the Gaussian WZ setup. The dashed horizontal lines are quantization boundaries, and the colors between boundaries represent unique values of $u$. We depict the decoding function as separate plots for each value of $u$, using the same color assignment. The visualized models on the left and right panels achieve $-15.44$ dB at $2.00$ bits and $-25.67$ dB at $2.78$ bits, respectively. Note that both of the visualized models achieve a rate--distortion (R-D) performance better than the point-to-point R-D function, due to many-to-one mapping functions recovered by the proposed solution.
  • ...and 4 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Definition 1
  • Definition 2
  • Definition 3
  • Definition 4