Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion

Anle Ke, Xu Zhang, Tong Chen, Ming Lu, Chao Zhou, Jiawen Gu, Zhan Ma

TL;DR

This work tackles ultra-low-rate image compression by coupling Semantic Residual Coding (SRC) with a Compression-aware Diffusion Model (CDM) in a framework named ResULIC. SRC leverages multimodal large models to extract a compact semantic residual that eliminates redundancy between the original image and its latent representation, while a Perceptual Fidelity Optimizer (Pfo) refines prompts to boost perceptual quality. CDM links diffusion steps to compression levels, enabling bitrate-aware sampling and efficient decoding without retraining, leading to substantial improvements in perceptual realism (FID/KID) and fidelity (LPIPS/DISTS) with notable BD-rate savings on standard benchmarks. Extensive experiments demonstrate robust gains across multiple MLLMs and datasets, with ablations confirming the contributions of SRC, Pfo, and CDM. Overall, ResULIC advances diffusion-based ultra-low-rate image compression by delivering higher-quality reconstructions at lower bitrates and with reduced decoding latency.

Abstract

Existing multimodal large model-based image compression frameworks often rely on a fragmented integration of semantic retrieval, latent compression, and generative models, resulting in suboptimal performance in both reconstruction fidelity and coding efficiency. To address these challenges, we propose a residual-guided ultra lowrate image compression named ResULIC, which incorporates residual signals into both semantic retrieval and the diffusion-based generation process. Specifically, we introduce Semantic Residual Coding (SRC) to capture the semantic disparity between the original image and its compressed latent representation. A perceptual fidelity optimizer is further applied for superior reconstruction quality. Additionally, we present the Compression-aware Diffusion Model (CDM), which establishes an optimal alignment between bitrates and diffusion time steps, improving compression-reconstruction synergy. Extensive experiments demonstrate the effectiveness of ResULIC, achieving superior objective and subjective performance compared to state-of-the-art diffusion-based methods with - 80.7%, -66.3% BD-rate saving in terms of LPIPS and FID. Project page is available at https: //njuvision.github.io/ResULIC/.

Ultra Lowrate Image Compression with Semantic Residual Coding and Compression-aware Diffusion

TL;DR

This work tackles ultra-low-rate image compression by coupling Semantic Residual Coding (SRC) with a Compression-aware Diffusion Model (CDM) in a framework named ResULIC. SRC leverages multimodal large models to extract a compact semantic residual that eliminates redundancy between the original image and its latent representation, while a Perceptual Fidelity Optimizer (Pfo) refines prompts to boost perceptual quality. CDM links diffusion steps to compression levels, enabling bitrate-aware sampling and efficient decoding without retraining, leading to substantial improvements in perceptual realism (FID/KID) and fidelity (LPIPS/DISTS) with notable BD-rate savings on standard benchmarks. Extensive experiments demonstrate robust gains across multiple MLLMs and datasets, with ablations confirming the contributions of SRC, Pfo, and CDM. Overall, ResULIC advances diffusion-based ultra-low-rate image compression by delivering higher-quality reconstructions at lower bitrates and with reduced decoding latency.

Abstract

Existing multimodal large model-based image compression frameworks often rely on a fragmented integration of semantic retrieval, latent compression, and generative models, resulting in suboptimal performance in both reconstruction fidelity and coding efficiency. To address these challenges, we propose a residual-guided ultra lowrate image compression named ResULIC, which incorporates residual signals into both semantic retrieval and the diffusion-based generation process. Specifically, we introduce Semantic Residual Coding (SRC) to capture the semantic disparity between the original image and its compressed latent representation. A perceptual fidelity optimizer is further applied for superior reconstruction quality. Additionally, we present the Compression-aware Diffusion Model (CDM), which establishes an optimal alignment between bitrates and diffusion time steps, improving compression-reconstruction synergy. Extensive experiments demonstrate the effectiveness of ResULIC, achieving superior objective and subjective performance compared to state-of-the-art diffusion-based methods with - 80.7%, -66.3% BD-rate saving in terms of LPIPS and FID. Project page is available at https: //njuvision.github.io/ResULIC/.
Paper Structure (39 sections, 3 theorems, 33 equations, 24 figures, 8 tables, 2 algorithms)

This paper contains 39 sections, 3 theorems, 33 equations, 24 figures, 8 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Framework Overview
  4. Semantic Residual Coding (SRC)
  5. Semantic residual retrieval (Srr)
  6. Perceptual fidelity optimization (Pfo)
  7. Compression-aware Diffusion Model (CDM)
  8. Modeling Diffusion Steps with Compression Levels
  9. Noise Adding Process
  10. Reverse Sampling Process
  11. Training Objective
  12. Experiments
  13. Experimental Setup
  14. Main Results
  15. Evaluation
  16. ...and 24 more sections

Key Result

Theorem 5.2

Given the distributions defined in Equation add, we have $z_n \perp z_{n-1} \mid z_0, z_c,$

Figures (24)

  • Figure 1: (a) The separate design for existing frameworks. The MLLM indicates the Multimodal Large Vision-Language Model. The latent coding represents the compression of features within the latent space. (b) Our pipeline with proposed Semantic Residual Coding and Compression-aware Diffusion Model. (c) Comparison with existing diffusion-based ultra lowrate image compression methods on CLIC2020 dataset.
  • Figure 2: Visual comparison at extremely low bitrates. The bitrate is averaged over samples for each tested method. The major structure on the left ($<$ 0.005 bpp) and the details of the text, hand, and clock on the right ($<$ 0.05 bpp) are better preserved.
  • Figure 3: ResULIC Overview: (1) The feature compressor transforms the original image $x$ into the compressed latent feature $z_c$. (2) The Semantic residual retrieval (Srr) generates optimized captions by analyzing both the decoded image $x'$ and the original $x$, with the plugin play module Perceptual fidelity optimizer (Pfo) to further improve reconstruction quality. (3) Text tokens are embedded into $c$ and combined with $z_c$ as conditions for the Compression-aware Diffusion Model (CDM) to generate the final image $x_r$.
  • Figure 4: The process of Semantic Residual Retrieval (Srr) involves implementing the MLLM to remove redundant text and correct inconsistencies.
  • Figure 5: Demonstration of Pfo optimized prompts and corresponding reconstruction. The prompts after optimization are human-readable with mixed real words and gibberish (non-word token sequences).
  • ...and 19 more figures

Theorems & Definitions (8)

  • Definition 5.1: Noise Addition Mechanism
  • Theorem 5.2: Conditional Independence of $z_n$ and $z_{n-1}$
  • proof
  • Definition 2.1: Noise Addition Mechanism
  • Theorem 2.2: Conditional Independence of $z_n$ and $z_{n-1}$
  • proof
  • Theorem 2.3: Parameter Relationships
  • proof