Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lossy Image Compression with Foundation Diffusion Models

Lucas Relic, Roberto Azevedo, Markus Gross, Christopher Schroers

TL;DR

This work forms the removal of quantization error as a denoising task, using diffusion to recover lost information in the transmitted image latent, enabling the use of foundation models as a strong prior without additional fine tuning of the backbone.

Abstract

Incorporating diffusion models in the image compression domain has the potential to produce realistic and detailed reconstructions, especially at extremely low bitrates. Previous methods focus on using diffusion models as expressive decoders robust to quantization errors in the conditioning signals, yet achieving competitive results in this manner requires costly training of the diffusion model and long inference times due to the iterative generative process. In this work we formulate the removal of quantization error as a denoising task, using diffusion to recover lost information in the transmitted image latent. Our approach allows us to perform less than 10% of the full diffusion generative process and requires no architectural changes to the diffusion model, enabling the use of foundation models as a strong prior without additional fine tuning of the backbone. Our proposed codec outperforms previous methods in quantitative realism metrics, and we verify that our reconstructions are qualitatively preferred by end users, even when other methods use twice the bitrate.

Lossy Image Compression with Foundation Diffusion Models

TL;DR

This work forms the removal of quantization error as a denoising task, using diffusion to recover lost information in the transmitted image latent, enabling the use of foundation models as a strong prior without additional fine tuning of the backbone.

Abstract

Incorporating diffusion models in the image compression domain has the potential to produce realistic and detailed reconstructions, especially at extremely low bitrates. Previous methods focus on using diffusion models as expressive decoders robust to quantization errors in the conditioning signals, yet achieving competitive results in this manner requires costly training of the diffusion model and long inference times due to the iterative generative process. In this work we formulate the removal of quantization error as a denoising task, using diffusion to recover lost information in the transmitted image latent. Our approach allows us to perform less than 10% of the full diffusion generative process and requires no architectural changes to the diffusion model, enabling the use of foundation models as a strong prior without additional fine tuning of the backbone. Our proposed codec outperforms previous methods in quantitative realism metrics, and we verify that our reconstructions are qualitatively preferred by end users, even when other methods use twice the bitrate.
Paper Structure (26 sections, 6 equations, 7 figures, 1 algorithm)

This paper contains 26 sections, 6 equations, 7 figures, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Neural Image Compression.
  5. Diffusion.
  6. Method
  7. Latent Diffusion Model Backbone
  8. Parameter Estimation
  9. Adaptive Quantization.
  10. Timestep Prediction.
  11. Architecture.
  12. Entropy Coding
  13. Optimization
  14. Experiments
  15. Datasets.
  16. ...and 11 more sections

Figures (7)

  • Figure 1: Visual examples of our proposed method and various classes of image compression codecs. Traditional (BPG BPG) and autoencoder-based (ELIC he2022elic) codecs suffer from blocking or blurring, and reconstructions from GAN-based (ILLM muckley2023Improving) and previous diffusion-based (HFD hoogeboom2023HighFidelity) methods contain high-frequency artifacts. Our proposal is as realistic as the original image while recovering a high level of detail. Bitrates are shown relative to our method. Best viewed digitally.
  • Figure 2: Rate-distortion (\ref{['subfig:naive_ldm_rd']}) and visual (\ref{['subfig:naive_ldm_vis']}) comparisons of our method to naively quantizing and entropy coding the latents of a latent diffusion model (Stable Diffusion rombach2022HighResolution). The LDM baseline requires nearly triple the bits to achieve comparable performance to our method and severly degrades the image at lower bitrates. Performing additional diffusion steps still does not produce a realistic image (\ref{['subfig:naive_ldm_vis']}, right). The color gradient of the dots in \ref{['subfig:naive_ldm_rd']} represents the number of denoising steps.
  • Figure 3: Overview of our approach. The input image $\mathbf{x}$ is encoded into latent space and transformed according to predicted parameters $\gamma$ before quantization and entropy coding. The quantized representation $\mathbf{\hat{z}}$ is transmitted with $\gamma$ and predicted diffusion timestep $t$ as side information. At the reciever the latent is inverse transformed, diffused over $t$ steps, and decoded back to image space.
  • Figure 4: Intermediate states of the sequential denoising process in our decoder. Our method predicts the optimal number of denoising steps, highlighted in red, to produce the most perceptually pleasing output. Best viewed digitally.
  • Figure 5: Quantitative comparison of our method with other baselines. We outperform all methods in (a) rate-realism while remaining competitive with the best performing generative codecs in (b) pixel-wise distortion metrics.
  • ...and 2 more figures