Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Distributed Compression with Learned Heegard-Berger Scheme

Eyyup Tasci, Ezgi Ozyilkan, Oguzhan Kubilay Ulger, Elza Erkip

TL;DR

This work addresses robust lossy distributed compression when decoder-side information may be unavailable, formulating neural schemes for the Heegard–Berger / Kaspi problem. It introduces three end-to-end learning frameworks (joint, marginal, conditional) that minimize variational upper bounds on the HB objective, using discrete neural encoders and decoders and, in the conditional variant, a Slepian–Wolf coder to enable high-dimensional binning. The results show that the learned compressors reproduce HB-appropriate strategies, with the marginal variant exhibiting binning-like behavior and the conditional model closely approaching the HB bound in the quadratic–Gaussian setting. Overall, the study demonstrates practical, one-shot robust distributed compression methods that adapt to side information availability and can extend to more complex, higher-dimensional sources.

Abstract

We consider lossy compression of an information source when decoder-only side information may be absent. This setup, also referred to as the Heegard-Berger or Kaspi problem, is a special case of robust distributed source coding. Building upon previous works on neural network-based distributed compressors developed for the decoder-only side information (Wyner-Ziv) case, we propose learning-based schemes that are amenable to the availability of side information. We find that our learned compressors mimic the achievability part of the Heegard-Berger theorem and yield interpretable results operating close to information-theoretic bounds. Depending on the availability of the side information, our neural compressors recover characteristics of the point-to-point (i.e., with no side information) and the Wyner-Ziv coding strategies that include binning in the source space, although no structure exploiting knowledge of the source and side information was imposed into the design.

Robust Distributed Compression with Learned Heegard-Berger Scheme

TL;DR

This work addresses robust lossy distributed compression when decoder-side information may be unavailable, formulating neural schemes for the Heegard–Berger / Kaspi problem. It introduces three end-to-end learning frameworks (joint, marginal, conditional) that minimize variational upper bounds on the HB objective, using discrete neural encoders and decoders and, in the conditional variant, a Slepian–Wolf coder to enable high-dimensional binning. The results show that the learned compressors reproduce HB-appropriate strategies, with the marginal variant exhibiting binning-like behavior and the conditional model closely approaching the HB bound in the quadratic–Gaussian setting. Overall, the study demonstrates practical, one-shot robust distributed compression methods that adapt to side information availability and can extend to more complex, higher-dimensional sources.

Abstract

We consider lossy compression of an information source when decoder-only side information may be absent. This setup, also referred to as the Heegard-Berger or Kaspi problem, is a special case of robust distributed source coding. Building upon previous works on neural network-based distributed compressors developed for the decoder-only side information (Wyner-Ziv) case, we propose learning-based schemes that are amenable to the availability of side information. We find that our learned compressors mimic the achievability part of the Heegard-Berger theorem and yield interpretable results operating close to information-theoretic bounds. Depending on the availability of the side information, our neural compressors recover characteristics of the point-to-point (i.e., with no side information) and the Wyner-Ziv coding strategies that include binning in the source space, although no structure exploiting knowledge of the source and side information was imposed into the design.
Paper Structure (12 sections, 1 theorem, 18 equations, 5 figures)

This paper contains 12 sections, 1 theorem, 18 equations, 5 figures.

Table of Contents

  1. Introduction
  2. Heegard--Berger Problem Setup
  3. General setting
  4. Quadratic-Gaussian case
  5. Operational Neural Heegard--Berger Schemes
  6. Discussion
  7. Conclusion
  8. The Minimum Distortion For Eq. \ref{['eq:expdist']}
  9. Proof of the Upper Bound in Eq. \ref{['eq:joint_upper_bound']}
  10. Operational Heegard--Berger Compressors Illustrated in Figs. \ref{['fig:unified_model']}, \ref{['fig:marginal_model']} and \ref{['fig:conditional_model']}
  11. Additional Details on Experimental Setup
  12. Additional Experiments

Key Result

Theorem 1

(Heegard--Berger Theorem Heegard_Berger85) Let $(X,Y)$ be a pair of random variables with joint distribution $p(x,y)$, representing the source and the correlated side information respectively, and $d_i : \mathcal{X} \times \hat{\mathcal{X}}_i \rightarrow \mathbb{R}_{\geq 0}$ be single letter distort where the minimization is over all conditional probability distributions $p(w, u \vert x)$ such tha

Figures (5)

  • Figure 1: Lossy source coding when side information may be absent, also known as the Heegard--Berger or the Kaspi problem.
  • Figure 2: The three lossy compression systems that we consider: (a) learned compressor sending a joint description for both decoders using a classic entropy coder (i.e., the joint formulation, see Eq. \ref{['eq:proposed_loss_joint']}), (b) learned compressors sending individual descriptions to both decoders using classic entropy coders (i.e., the marginal formulation, see Eq. \ref{['eq:proposed_loss_marginal']}), and (c) using a combination of a classic entropy coder and an ideal Slepian--Wolf coder (i.e., the conditional formulation, see Eq. \ref{['eq:proposed_loss_conditional']}). The learned parameters are indicated in green.
  • Figure 3: Visualizations of the learned optimized encoders, $w = \mathop{\mathrm{arg\,max}}\limits_{h} p_{\boldsymbol{\omega}}(h|x)$ and $u = \mathop{\mathrm{arg\,max}}\limits_{l} p_{\boldsymbol{\gamma}}(l|w, x)$ in Eq. \ref{['eq:upper_bound_sep_marg']}, and of the decoders, $\hat{x}_{1} = g_{\boldsymbol{\kappa}}(w)$ and $\hat{x}_{2}= g_{\boldsymbol{\iota}}(w, u, y)$ in Eq \ref{['eq:d_s']}, for the marginal formulation (see Fig. \ref{['fig:marginal_model']} and Eq. \ref{['eq:proposed_loss_marginal']}). Left: The dashed vertical red lines are quantization boundaries induced by $w$, and the codebook points represent the outputs of the decoder $g_{\boldsymbol{\kappa}}$ to which all source values within the corresponding quantization region are mapped. The height of each codebook stalk represents the likelihood of that code vector under the entropy model $q_{{\boldsymbol{\zeta}}}$ (see Eq. \ref{['eq:upper_bound_sep_marg']}). Right: The dashed horizontal red lines coincide with those depicted in the left panel induced by $w$, while the dotted-dashed horizontal blue lines are quantization boundaries induced by $u$. The colors between each boundary represent a unique $(w,u)$ pair. We depict the decoding function learned by $g_{\boldsymbol{\iota}}$ with the solid lines, each representing a different pair of $(w,u)$ as inputs within its respective quantization region, distinguished by unique colors. The experimental setup parameters (see Section \ref{['subsec:experimental_setup']}) are configured as $\sigma^{2}_x=1.00$, $\sigma^2_n = 0.01$ and $\beta = 0.20$. This model attains -15.55 dB at a rate of 2.85 bits (see Fig. \ref{['fig:RD_var_0.1']} in Appendix \ref{['sec:appendix_experiments']}).
  • Figure 4: Rate--distortion (R-D) performances obtained with joint, marginal and conditional formulations (see Eqs. \ref{['eq:proposed_loss_joint']}, \ref{['eq:proposed_loss_marginal']} and \ref{['eq:proposed_loss_conditional']}), where experimental setup parameters (see Section \ref{['subsec:experimental_setup']}) are set as $\sigma^{2}_{x}=1.00$, $\sigma^{2}_{n} = 0.10$ and $\beta = 0.01$. In both panels, we plot the empirical results versus the asymptotic bounds. We provide the expected distortion achieved by two decoders (see Fig. \ref{['fig:sys']}) in the left panel, while we only plot the distortion attained by the informed decoder, which has access to side information, in the right panel. The 1.53 dB refers to the mean-squared error gap that the entropy-constrained one-shot lattice quantizer is subjected to in high-rate regime quantization.
  • Figure 5: Rate--distortion (R-D) performances obtained with joint, marginal and conditional formulations (see Eqs. \ref{['eq:proposed_loss_joint']}, \ref{['eq:proposed_loss_marginal']} and \ref{['eq:proposed_loss_conditional']}), where experimental setup parameters (see Section \ref{['subsec:experimental_setup']}) are set as $\sigma^{2}_{x}=1.00$, $\sigma^{2}_{n} = 0.01$ and $\beta = 0.20$. In both panels, we plot the empirical results versus the asymptotic bounds. We provide the expected distortion achieved by two decoders (see Fig. \ref{['fig:sys']}) in the left panel, while we only plot the distortion attained by the informed decoder, which has access to side information, in the right panel. The 1.53 dB refers to the mean-squared error gap that the entropy-constrained one-shot lattice quantizer is subjected to in high-rate regime quantization.

Theorems & Definitions (1)

  • Theorem 1