Higher fidelity perceptual image and video compression with a latent conditioned residual denoising diffusion model

TL;DR

This work targets the fidelity gap in perceptual diffusion-based compression by introducing a hybrid end-to-end codec that combines a fidelity-optimized decoder with a latent-conditioned residual diffusion model (ResCDC). The diffusion model refines the decoder reconstruction by predicting residual details, achieving up to about +2 dB PSNR improvements while maintaining competitive perceptual metrics such as LPIPS and FID on standard benchmarks, and extending naturally to video. Training optimizes a joint loss that balances bitrate, distortion, perceptual quality, and decoder fidelity, while decoding leverages a DDIM-like sampler with a limited number of steps to control latency. The approach demonstrates strong gains over CDC and is adaptable to video compression, offering practical improvements in perceptual fidelity at the cost of slower decoding, with clear ablations guiding parameter choices and step counts.

Abstract

Denoising diffusion models achieved impressive results on several image generation tasks often outperforming GAN based models. Recently, the generative capabilities of diffusion models have been employed for perceptual image compression, such as in CDC. A major drawback of these diffusion-based methods is that, while producing impressive perceptual quality images they are dropping in fidelity/increasing the distortion to the original uncompressed images when compared with other traditional or learned image compression schemes aiming for fidelity. In this paper, we propose a hybrid compression scheme optimized for perceptual quality, extending the approach of the CDC model with a decoder network in order to reduce the impact on distortion metrics such as PSNR. After using the decoder network to generate an initial image, optimized for distortion, the latent conditioned diffusion model refines the reconstruction for perceptual quality by predicting the residual. On standard benchmarks, we achieve up to +2dB PSNR fidelity improvements while maintaining comparable LPIPS and FID perceptual scores when compared with CDC. Additionally, the approach is easily extensible to video compression, where we achieve similar results.

Figures (9)

• Figure 1: Overview of the proposed architecture
• Figure 2: Qualitative comparison between CDC and ResCDC (ours)
• Figure 3: Results on the Kodak images. CDC $\epsilon$ models were sampled using 500 steps, CDC $x_0$ models using 17 steps, ResCDC (ours) models using 100 steps.
• Figure 4: Results on the DIV2K validation-set. CDC $\epsilon$ models were sampled using 500 steps, CDC $x_0$ models using 17 steps, ResCDC (ours) models using 100 steps.
• Figure 5: Ablation comparing the impact of the amount of sampling steps on the Kodak images. The reference CDC $\epsilon$ was sampled using 500 steps.