A Theoretical Framework for Rate-Distortion Limits in Learned Image Compression

TL;DR

This work proposes a principled framework to quantify the rate-distortion limits of learned image compression by decomposing the practical gap into variance estimation, quantization strategy, and context modeling. By adopting a Gaussian latent model and a Gaussian test channel, it derives theoretical limits and constructs joint simulations that incorporate optimal variance, Gaussian quantization, and autoregressive mean prediction to approximate the true R-D bound. The approach yields interpretable insights into where current learned codecs fall short and how to design architectures that close the gap, demonstrated through ablations and comparisons with prior estimators and practical codecs. The resulting framework provides a actionable benchmark for evaluating and guiding future improvements in neural image compression research.

Abstract

We present a novel systematic theoretical framework to analyze the rate-distortion (R-D) limits of learned image compression. While recent neural codecs have achieved remarkable empirical results, their distance from the information-theoretic limit remains unclear. Our work addresses this gap by decomposing the R-D performance loss into three key components: variance estimation, quantization strategy, and context modeling. First, we derive the optimal latent variance as the second moment under a Gaussian assumption, providing a principled alternative to hyperprior-based estimation. Second, we quantify the gap between uniform quantization and the Gaussian test channel derived from the reverse water-filling theorem. Third, we extend our framework to include context modeling, and demonstrate that accurate mean prediction yields substantial entropy reduction. Unlike prior R-D estimators, our method provides a structurally interpretable perspective that aligns with real compression modules and enables fine-grained analysis. Through joint simulation and end-to-end training, we derive a tight and actionable approximation of the theoretical R-D limits, offering new insights into the design of more efficient learned compression systems.

Figures (6)

• Figure 1: Architecture of the Proposed Rate-Distortion Simulation Framework
• Figure 2: Top (a): Rate-variance curves under uniform quantization and the optimal Gaussian test channel. Bottom (b): Pointwise difference in coding rate between the two strategies, highlighting the inefficiency of uniform quantization.
• Figure 3: From left to right: original image, latents, uniform-quantized output, and Gaussian test-channel output. Low-variance regions (e.g., sky) receive zero-rate allocation only under the Gaussian test channel.
• Figure 4: Rate-distortion comparison of different components. Left (a): Comparison between the Hyperprior baseline and variants enhanced with (i) optimal quantization (Gaussian test channel), (ii) optimal variance estimation, and (iii) joint optimization of both. Right (b): Comparison between Hyperprior baseline, the context-enhanced minnen2018joint model, joint optimization without and with context modeling.
• Figure 5: Left (a): Comparison of our theoretical R-D estimates with prior sample-driven estimators and the actual performance of learned image compression models on the Kodak dataset measured by PSNR. Middle (b): Comparison of ours with practical compression models measured by MS-SSIM. Right (c): Comparison of different R-D estimation methods on the MNIST training set.

Theorems & Definitions (5)

• Theorem 1
• proof
• Theorem 2
• proof
• proof : Proof (from Gaussian test channel perspective)