Data-Driven Neural Estimation of Indirect Rate-Distortion Function

TL;DR

The paper addresses estimating the indirect rate-distortion function (iRDF) when the encoder observes $X$ correlated with the source $S$ rather than $S$ itself. It shows that the iRDF can be computed as the RDF with a reduced distortion $\bar{d}(x,y)=E[d(S,y)|X=x]$ and derives a dual variational formulation with $F(\lambda)$ and an optimal $P^*_{Y|X}$, enabling a curve sweep over $\lambda$ to obtain $R(D)$. To handle data-driven settings with unknown $P_{S,X}$, the authors propose NEIRD, a neural estimator that combines a MMSE-based estimator $g_{\hat{\theta}}(X,Y)$ for $\bar{d}$ and a neural mapping $T_{\theta}$ to model $Q_Y$, using empirical objectives to approximate the iRDF. They prove consistency and demonstrate accuracy on synthetic data and MNIST, indicating practical viability for applications like remote sensing and goal-oriented communication, where explicit joint statistics are unavailable.

Abstract

The rate-distortion function (RDF) has long been an information-theoretic benchmark for data compression. As its natural extension, the indirect rate-distortion function (iRDF) corresponds to the scenario where the encoder can only access an observation correlated with the source, rather than the source itself. Such scenario is also relevant for modern applications like remote sensing and goal-oriented communication. The iRDF can be reduced into a standard RDF with the distortion measure replaced by its conditional expectation conditioned upon the observation. This reduction, however, leads to a non-trivial challenge when one needs to estimate the iRDF given datasets only, because without statistical knowledge of the joint probability distribution between the source and its observation, the conditional expectation cannot be evaluated. To tackle this challenge, starting from the well known fact that conditional expectation is the minimum mean-squared error estimator and exploiting a Markovian relationship, we identify a functional equivalence between the reduced distortion measure in the iRDF and the solution of a quadratic loss minimization problem, which can be efficiently approximated by neural network approach. We proceed to reformulate the iRDF as a variational problem corresponding to the Lagrangian representation of the iRDF curve, and propose a neural network based approximate solution, integrating the aforementioned distortion measure estimator. Asymptotic analysis guarantees consistency of the solution, and numerical experimental results demonstrate the accuracy and effectiveness of the algorithm.

Figures (3)

• Figure 1: Indirect rate-distortion model
• Figure 2: Performance of NEIRD on synthetic datasets.
• Figure 3: Performance of NEIRD on MNIST.

Theorems & Definitions (4)

• Proposition 1
• proof
• Theorem 2
• proof