Towards Efficient Low-rate Image Compression with Frequency-aware Diffusion Prior Refinement

TL;DR

This work tackles the challenge of high-quality image reconstruction at ultra-low bitrates by leveraging diffusion priors within latent compression. It introduces a Consistency Refinement Estimator (CRE) with Frequency-aware Skip Estimation (FaSE) and Frequency Decoupling Attention (FDA) to align diffusion priors with compressed latents and to enable fast two-step decoding without updating the backbone model. A hybrid control scheme combines image-level and semantic guidance to stabilize and improve reconstruction, while a two-stage training strategy optimizes both rate–distortion and diffusion-sampling patterns. Empirically, DiffCR achieves substantial BD-rate savings and significant speed-ups over state-of-the-art diffusion-based baselines across standard datasets, demonstrating a practical pathway to diffusion-guided, real-time image compression at very low bitrates.

Abstract

Recent advancements in diffusion-based generative priors have enabled visually plausible image compression at extremely low bit rates. However, existing approaches suffer from slow sampling processes and suboptimal bit allocation due to fragmented training paradigms. In this work, we propose Accelerate \textbf{Diff}usion-based Image Compression via \textbf{C}onsistency Prior \textbf{R}efinement (DiffCR), a novel compression framework for efficient and high-fidelity image reconstruction. At the heart of DiffCR is a Frequency-aware Skip Estimation (FaSE) module that refines the $ε$-prediction prior from a pre-trained latent diffusion model and aligns it with compressed latents at different timesteps via Frequency Decoupling Attention (FDA). Furthermore, a lightweight consistency estimator enables fast \textbf{two-step decoding} by preserving the semantic trajectory of diffusion sampling. Without updating the backbone diffusion model, DiffCR achieves substantial bitrate savings (27.2\% BD-rate (LPIPS) and 65.1\% BD-rate (PSNR)) and over $10\times$ speed-up compared to SOTA diffusion-based compression baselines.

Figures (6)

• Figure 1: Qualitative and quantitative comparison between DiffEIC, Perco, HiFiC, MS-ILLM, CDC, and our proposed approach. DiffCR (Ours) not only possesses realism but also faithfully restores the details of the original image. The radar chart in the bottom right corner illustrates the performance comparison among various methods on the CLIC20 dataset. Best viewed when zoomed in.
• Figure 2: Illustration of the proposed DiffCR. RRDB signifies Residual in Residual Dense Block wang2018esrgan. DiffCR utilizes two branches to guide pre-trained denoising networks in generation and reinforces prior correction and refinement through the FaSE module, aligning it with the reconstruction objective of the compressor. Unlike the ordinary sampling process (depicted by the grey dashed line), DiffCR permits a two-step sampling approach (illustrated by the blue-purple arrows).
• Figure 3: (Left) The trend of frequency energy variation in reconstruction by diffusion models at different time steps. (Right) Quantitative performance comparison of the compressor, denoising network, and FaSE in proceeding $\boldsymbol{z_0}$-prediction.
• Figure 4: Visualization of bit-rate allocation on the Kodak.
• Figure 5: Illustration of the FDA module, where $Q_{\cdot}K_{\cdot}V_{\cdot}$ corresponds to Query, Key, and Value in the attention mechanism.