Neural Image Compression with Text-guided Encoding for both Pixel-level and Perceptual Fidelity

TL;DR

The paper tackles the conflict between pixel-level fidelity and perceptual quality in text-guided image compression. It introduces TACO, an encoder-centric approach that augments a PSNR-oriented backbone with a CLIP-based text adapter to inject semantic information directly into the latent representation, and it trains with a joint image-text loss to align reconstructions with caption semantics. Key contributions include (i) a cross-attention-based text adapter injected into the encoder, (ii) a joint loss that combines rate, distortion, LPIPS, and CLIP-based text-image alignment, and (iii) strong LPIPS performance at standard bitrates with competitive PSNR across multiple datasets while preserving textual content. The results demonstrate that text can meaningfully improve perceptual quality without substantial sacrifices in pixel accuracy, offering a practical pathway for text-conditioned image compression that avoids the diversity of text-guided decoding.

Abstract

Recent advances in text-guided image compression have shown great potential to enhance the perceptual quality of reconstructed images. These methods, however, tend to have significantly degraded pixel-wise fidelity, limiting their practicality. To fill this gap, we develop a new text-guided image compression algorithm that achieves both high perceptual and pixel-wise fidelity. In particular, we propose a compression framework that leverages text information mainly by text-adaptive encoding and training with joint image-text loss. By doing so, we avoid decoding based on text-guided generative models -- known for high generative diversity -- and effectively utilize the semantic information of text at a global level. Experimental results on various datasets show that our method can achieve high pixel-level and perceptual quality, with either human- or machine-generated captions. In particular, our method outperforms all baselines in terms of LPIPS, with some room for even more improvements when we use more carefully generated captions.

Figures (17)

• Figure 1: Pixel-wise fidelity vs. perceptual fidelity, at 0.40 bpp. We compare pixel-wise and perceptual fidelity of image compression codecs on MS-COCO 30k. The proposed TACO achieves competitive results in both metrics. The reported figures for Qin 2023 has been measured on MS-COCO 40k, and some figures have been interpolated from nearest bpp models ($\medbullet$: PSNR-focused, $\blacksquare$: perception-focused, $\blacktriangle$: text-guided, $\times$: handcrafted).
• Figure 2: Qualitative results. We compare TACO against MS-ILLM and LIC-TCM, which focuses on perception and PSNR, respectively. TACO uses slightly less bpp than baselines. Comparing with MS-ILLM, TACO tends to suffer less from hallucinated artifacts (see teeth). Comparing with LIC-TCM, TACO can reconstruct sharper details (see lips). See \ref{['app:add_qualitative_examples']} for more examples.
• Figure 3: Text-guided decoding strategies vs. TACO. (Top) Text-guided decoding with diffusion-based decoders careil24. (Middle) Text-guided decoder utilizing GAN qin_comp_23. (Bottom) TACO is a much simpler yet effective strategy.
• Figure 4: Text adapter of TACO. The text adapter first extracts features from the image caption using the CLIP text encoder. The textual features are then injected into the ELIC encoder through multiple cross-attention layers, interleaved with linear layers.
• Figure 5: Compression results: MS-COCO 30k. TACO achieves the best or competitive results in all metrics. In particular, TACO achieves the best LPIPS among all methods considered, while achieving only $\sim$1dB less PSNR than LIC-TCM. PerCo achieves an impressive FID, but TACO outperforms in terms of both PSNR and LPIPS. We also note a strange under-performance of HiFiC in FID under this setup. This may be due to the resizing operation when measuring FID.