Leveraging Diffusion Knowledge for Generative Image Compression with Fractal Frequency-Aware Band Learning

TL;DR

The paper addresses the challenge of balancing bitrate, distortion, and realism in learned image compression. It introduces FFAB-IC, a diffusion-informed framework that combines Fractal Frequency-Aware Band blocks with a pre-trained diffusion prior to guide encoding and decoding, and enforces frequency- and content-aware regularization to improve texture realism. The approach yields state-of-the-art performance on standard benchmarks, with significant BD-rate gains in DISTS and FID across Kodak, DIV2K, and MS-COCO 30K, while maintaining competitive encoding times. A key limitation is the slow diffusion-based processing, which the authors propose to accelerate in future work.

Abstract

By optimizing the rate-distortion-realism trade-off, generative image compression approaches produce detailed, realistic images instead of the only sharp-looking reconstructions produced by rate-distortion-optimized models. In this paper, we propose a novel deep learning-based generative image compression method injected with diffusion knowledge, obtaining the capacity to recover more realistic textures in practical scenarios. Efforts are made from three perspectives to navigate the rate-distortion-realism trade-off in the generative image compression task. First, recognizing the strong connection between image texture and frequency-domain characteristics, we design a Fractal Frequency-Aware Band Image Compression (FFAB-IC) network to effectively capture the directional frequency components inherent in natural images. This network integrates commonly used fractal band feature operations within a neural non-linear mapping design, enhancing its ability to retain essential given information and filter out unnecessary details. Then, to improve the visual quality of image reconstruction under limited bandwidth, we integrate diffusion knowledge into the encoder and implement diffusion iterations into the decoder process, thus effectively recovering lost texture details. Finally, to fully leverage the spatial and frequency intensity information, we incorporate frequency- and content-aware regularization terms to regularize the training of the generative image compression network. Extensive experiments in quantitative and qualitative evaluations demonstrate the superiority of the proposed method, advancing the boundaries of achievable distortion-realism pairs, i.e., our method achieves better distortions at high realism and better realism at low distortion than ever before.

Figures (7)

• Figure 1: Motivation and superiority. The visual results show the directional frequency bands captured through the designed FFAB block. Each frequency band demonstrates distinct characteristics, allowing for an intuitive understanding of the inherent fractal patterns within natural image. (b) Our FFAB-IC network could deliver a significant performance improvement on the Kodak dataset in terms of DISTS value.
• Figure 2: Comparison of our generative image compression technique with existing methods. (a) Previous frequency-domain image processing methods (e.g.,wang2023fourlliepaul2024f2formerli2024frequencyaware) analyze the spectrum. (b) The FFAB-IC network utilizes frequency fractal bands, including both isotropic and anisotropic features, while embedding an injected diffusion prior into the compression framework.
• Figure 3: Overview of the Proposed Fractal Frequency-Aware Band Image Compression (FFAB-IC) Framework. Our FFAB-IC framework leverages a pre-trained Stable Diffusion, integrating generative prior knowledge with Fractal Frequency-Aware Band information to enhance contextual understanding in two key aspects: (a) Feature Level Operation: the framework enables window attention interactions across low-frequency, high-frequency, vertical, and horizontal bands. This allows for effective integration between generative features and image features in the representation space. (b) Optimization-Level Guidance: we introduce frequency domain constraints into the computation of content representation.
• Figure 4: Illustration of DISTS $\downarrow$ and FID $\downarrow$ results based on the evaluation metrics outlined in \ref{['Experiment Settings']}. More results can be found in the supplementary material. The proposed compression method (bottom left) significantly outperforms the baseline methods. Zooming in on the figure will provide a better look at the RD curve comparison results.
• Figure 5: Visual comparison of compression methods (a) VTM bross2021overview, (b) HiFiC mentzer2020high, (c) CRDR iwai2024controlling, and (d) Ours. Our method could effectively maintains fine details and improve overall image quality, as evidenced by the clarity in zoomed-in areas.