The Gap Between Principle and Practice of Lossy Image Coding

TL;DR

This work investigates why practical learned lossy image codecs do not reach Shannon's rate-distortion bound and decomposes the gap into five effects: modeling, approximation, amortization, digitization, and asymptotic. It formalizes a latent-based optimization framework with Lagrangian relaxation, analyzes each gap source, and demonstrates that reducing amortization, improving entropy modeling, and addressing digitization through per-sample optimization can yield meaningful rate savings. By estimating RD upper bounds with continuous stochastic models and evaluating on Kodak, the study shows potential RD improvements up to about 35% over VVC, alongside substantial empirical gains (up to 24%) with more capable transforms and entropy models. The findings highlight the high potential of future lossy image coding technologies and provide a roadmap for narrowing the theory-practice gap through architectural and optimization enhancements.

Abstract

Lossy image coding is the art of computing that is principally bounded by the image's rate-distortion function. This bound, though never accurately characterized, has been approached practically via deep learning technologies in recent years. Indeed, learned image coding schemes allow direct optimization of the joint rate-distortion cost, thereby outperforming the handcrafted image coding schemes by a large margin. Still, it is observed that there is room for further improvement in the rate-distortion performance of learned image coding. In this article, we identify the gap between the ideal rate-distortion function forecasted by Shannon's information theory and the empirical rate-distortion function achieved by the state-of-the-art learned image coding schemes, revealing that the gap is incurred by five different effects: modeling effect, approximation effect, amortization effect, digitization effect, and asymptotic effect. We design simulations and experiments to quantitively evaluate the last three effects, which demonstrates the high potential of future lossy image coding technologies.

Figures (5)

• Figure 1: (a) Illustration of a virtual channel $P_{\hat{X}|X}$ for lossy source coding; $X$ is the source and $\hat{X}$ is the reconstruction. (b) Illustration of instantiating $P_{\hat{X}|X}$ by a latent-based model; $Y$ refers to the latents.
• Figure 2: The compression efficiency of the estimated and empirical rate-distortion functions with different models and different numbers of parameters. A negative BD-rate indicates rate saving.
• Figure 3: The compression efficiency gain provided by the per-sample optimization on each model.
• Figure 4: The compression efficiency gap between the estimated and the empirical rate-distortion functions for different models and different numbers of parameters.
• Figure 5: Rate-distortion curves of different methods. The empirical and estimated RD shown in this figure is the performance of the Res-context model.