Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior

TL;DR

The paper tackles the mismatch between image codecs optimized for human perception and those tailored for machine analytics by introducing Diff-ICMH, a diffusion-prior–based compression framework. It jointly preserves semantic information via Semantic Consistency loss and enhances perceptual realism through generative priors, guided by a lightweight Tag Guidance Module. The bitstream encodes latent diffusion-space representations and semantic tags, with reconstruction performed by a pre-trained diffusion model conditioned on these signals. Extensive experiments across traditional vision tasks, multimodal retrieval, and open-set segmentation demonstrate strong generalization and state-of-the-art perceptual quality at diverse bitrates, achieved without task-specific adaptation.

Abstract

Image compression methods are usually optimized isolatedly for human perception or machine analysis tasks. We reveal fundamental commonalities between these objectives: preserving accurate semantic information is paramount, as it directly dictates the integrity of critical information for intelligent tasks and aids human understanding. Concurrently, enhanced perceptual quality not only improves visual appeal but also, by ensuring realistic image distributions, benefits semantic feature extraction for machine tasks. Based on this insight, we propose Diff-ICMH, a generative image compression framework aiming for harmonizing machine and human vision in image compression. It ensures perceptual realism by leveraging generative priors and simultaneously guarantees semantic fidelity through the incorporation of Semantic Consistency loss (SC loss) during training. Additionally, we introduce the Tag Guidance Module (TGM) that leverages highly semantic image-level tags to stimulate the pre-trained diffusion model's generative capabilities, requiring minimal additional bit rates. Consequently, Diff-ICMH supports multiple intelligent tasks through a single codec and bitstream without any task-specific adaptation, while preserving high-quality visual experience for human perception. Extensive experimental results demonstrate Diff-ICMH's superiority and generalizability across diverse tasks, while maintaining visual appeal for human perception. Code is available at: https://github.com/RuoyuFeng/Diff-ICMH.

Figures (14)

• Figure 1: Comparison of compression and reconstruction workflows between different methods.
• Figure 2: Illustration of the information transformation during compression processes in signal fidelity-oriented image compression and Diff-ICMH.
• Figure 3: Variation in the difference ($1$ minus cosine similarity) between features extracted from reconstructed images by different codecs and those from original images when input to pre-trained ResNet50 he2016deep, shown against increasing network depth.
• Figure 4: Overview of Diff-ICMH. Diff-ICMH consists of two parts: (left) image encoding/decoding and tag extraction; (right) condition-based image reconstruction. For simplicity, skip connections are omitted in the diagram.
• Figure 5: (a) Semantic Consistency loss. The latent variable $\mathbf{z}$ and the decoded latent variable $\hat{\mathbf{z}}$ are projected through the pre-trained diffusion model, resulting semantic representations that are then aligned. (b) Tag Guidance Module. Tags are extracted via tag extractor and losslessly compressed.