Generative Latent Coding for Ultra-Low Bitrate Image Compression

TL;DR

GLC proposes a generative latent coding framework that performs transform coding in the latent space of a VQ-VAE instead of pixel space to improve perceptual fidelity at ultra-low bitrates. It introduces a categorical hyper module and a code-prediction-based latent supervision to reduce bitrate and enhance semantic consistency. The approach achieves state-of-the-art results on natural and facial images, with less than 0.04 bpp on CLIC2020 and less than 0.01 bpp on CelebAHQ, and 45% bitrate reduction at the same FID on CLIC2020. The latent space enables flexible applications such as image restoration and style transfer, illustrating practical benefits beyond compression. Limitations include generalization to screen content, pointing to future work.

Abstract

Most existing image compression approaches perform transform coding in the pixel space to reduce its spatial redundancy. However, they encounter difficulties in achieving both high-realism and high-fidelity at low bitrate, as the pixel-space distortion may not align with human perception. To address this issue, we introduce a Generative Latent Coding (GLC) architecture, which performs transform coding in the latent space of a generative vector-quantized variational auto-encoder (VQ-VAE), instead of in the pixel space. The generative latent space is characterized by greater sparsity, richer semantic and better alignment with human perception, rendering it advantageous for achieving high-realism and high-fidelity compression. Additionally, we introduce a categorical hyper module to reduce the bit cost of hyper-information, and a code-prediction-based supervision to enhance the semantic consistency. Experiments demonstrate that our GLC maintains high visual quality with less than 0.04 bpp on natural images and less than 0.01 bpp on facial images. On the CLIC2020 test set, we achieve the same FID as MS-ILLM with 45% fewer bits. Furthermore, the powerful generative latent space enables various applications built on our GLC pipeline, such as image restoration and style transfer. The code is available at https://github.com/jzyustc/GLC.

Figures (22)

• Figure 1: Generative latent space of VQ-VAE exhibits better alignment with human perception than pixel space for ultra-low bitrate compression. Under comparable distortion levels (measured by signal-to-noise ratio, SNR), latent-space compression produces reconstructions with superior perceptual quality (measured by DISTS ding2020image) than pixel-space generative codec MS-ILLM muckley2023improving, as the compressed latents remain in the same latent code space.
• Figure 2: A qualitative comparison between HiFiC mentzer2020high, MS-ILLM muckley2023improving, Text+Sketch lei2023text+ and the proposed GLC. GLC produces images with high visual quality, even in regions with complex texture. In contrast, HiFiC and MS-ILLM exhibit noticeable artifacts, and Text+Sketch generates results that deviate significantly from the input. Best viewed when zoomed in.
• Figure 3: Illustration of the proposed Generative Latent Coding (GLC) framework. (Left) GLC firstly encodes the image into a generative latent representation (Section \ref{['generative_latent_auto_encoder']}), then compresses the latent with transform coding (Section \ref{['generative_latent_transform_coding']}), and finally reconstructs image from the compressed latent. (Right) We progressively train GLC in three stages (Section \ref{['progressive_training']}): In stage I, we train a generative VQ-VAE to obtain a human-perception-aligned latent space. In stage II, the transform coding module learns to compress the latent with a code-prediction-based latent supervision (Figure \ref{['fig:Latent_Supervision']}). Finally, in stage III, the entire network is fine-tuned jointly with a code-prediction-based pixel supervision to further enhance the compression performance.
• Figure 4: Illustration of the transform coding in latent space. (a) The model structure of the transform coding module. We further compare it with other coding schemes in operational diagrams : (b) indices-map coding mao2023extremejiang2023adaptivejiang2023face, (c) transform coding with factorized hyper module balle2018variationalhe2021checkerboardli2023neural and (d) proposed transform coding with categorical hyper module. Here AE and AD denote arithmetic encoding and decoding, VQ-E and VQ-D stand for VQ-indices-map encoding and decoding, Q refers to scalar quantization, U signifies the addition of uniform noise as a differential simulation of Q, and S denotes the spatial context entropy module.
• Figure 5: Example of comparison between factorized and categorical hyper modules. The proposed categorical $z$ encodes essential semantic and structural information with much less bits.