Neural Estimation of the Rate-Distortion Function With Applications to Operational Source Coding

TL;DR

This work tackles the challenge of estimating the rate-distortion function $R(D)$ for real-world data, where traditional Blahut–Arimoto methods fail to scale. It introduces NERD, a neural estimator that parameterizes the RD-achieving reproduction marginal via a neural generator and optimizes a min–max objective derived from a dual formulation, with strong consistency under mild conditions. Empirically, NERD accurately estimates $R(D)$ on Gaussian and real image datasets, and DNN compressors operate within a few bits of the estimated limit, suggesting near-optimal practical performance. Furthermore, the paper shows how to recover the RD-optimal channel and implement one-shot lossy compression via Reverse Channel Coding (PFR/ORC), yielding compression schemes with guarantees on rate and distortion and competitive results against DNN codecs. The work also discusses limitations, including sample complexity and scalability, and outlines future directions toward block coding and improved RCC methodologies.

Abstract

A fundamental question in designing lossy data compression schemes is how well one can do in comparison with the rate-distortion function, which describes the known theoretical limits of lossy compression. Motivated by the empirical success of deep neural network (DNN) compressors on large, real-world data, we investigate methods to estimate the rate-distortion function on such data, which would allow comparison of DNN compressors with optimality. While one could use the empirical distribution of the data and apply the Blahut-Arimoto algorithm, this approach presents several computational challenges and inaccuracies when the datasets are large and high-dimensional, such as the case of modern image datasets. Instead, we re-formulate the rate-distortion objective, and solve the resulting functional optimization problem using neural networks. We apply the resulting rate-distortion estimator, called NERD, on popular image datasets, and provide evidence that NERD can accurately estimate the rate-distortion function. Using our estimate, we show that the rate-distortion achievable by DNN compressors are within several bits of the rate-distortion function for real-world datasets. Additionally, NERD provides access to the rate-distortion achieving channel, as well as samples from its output marginal. Therefore, using recent results in reverse channel coding, we describe how NERD can be used to construct an operational one-shot lossy compression scheme with guarantees on the achievable rate and distortion. Experimental results demonstrate competitive performance with DNN compressors.

Figures (19)

• Figure 1: Inaccuracy of discretized Blahut-Arimoto in comparison to our method, $\widehat{R_\Theta(D)}_n$, for computing the rate distortion curve on the MNIST dataset. DNN compressors provide codes that lie in the achievable region. See text for details.
• Figure 2: Estimated $\widehat{R_\Theta(D)}_n$ (NERD) on Gaussian data ($m=20$, $\sigma_k^2 = 4e^{-\frac{1}{16}k}$) compared with DNN compressors.
• Figure 3: Estimated $\widehat{R_\Theta(D)}_n$ (NERD) of SVHN images vs. DNN compressors.
• Figure 4: Samples from trained $Y$-marginal $Q^*_Y$ (MNIST). As $R \rightarrow 0$, $Q^*_Y$ generates the mean image, achieving $D=D_{\max}$.
• Figure 5: Samples from trained $Y$-marginal $Q^*_Y$ (SVHN). As $R \rightarrow 0$, $Q^*_Y$ generates the mean image, achieving $D=D_{\max}$.

Theorems & Definitions (11)

• Lemma 1: Double-Minimization Form, cf. CoverThomas, Yeung2002AFC
• Lemma 2: Rate Function Duality, Dembo
• Definition 1: Neural Estimator of the Rate-Distortion Function (NERD)
• Theorem 1: Strong consistency of NERD
• Proposition 1
• Remark 1
• proof
• Theorem 2: Theorem \ref{['thm:strong_consistency']} in text
• proof
• Lemma 3