Extreme Generative Image Compression by Learning Text Embedding from Diffusion Models

TL;DR

The paper tackles the challenge of extremely low-bitrate image compression by encoding images as short textual embeddings learned through textual inversion and reconstructing them with a pre-trained text-to-image diffusion model. It introduces compression guidance, combining a small reference image cue with classifier-free guidance to steer diffusion toward faithful yet diverse reconstructions at around 0.07 bpp. The approach leverages a latent diffusion backbone (Stable Diffusion) and does not require training dedicated diffusion models, achieving superior perceptual metrics on Kodak while maintaining a near-constant bitrate. Computational cost is noted as a trade-off, with iterative embedding optimization and many denoising steps during generation, suitable for bandwidth-constrained scenarios where quality and diversity matter.

Abstract

Transferring large amount of high resolution images over limited bandwidth is an important but very challenging task. Compressing images using extremely low bitrates (<0.1 bpp) has been studied but it often results in low quality images of heavy artifacts due to the strong constraint in the number of bits available for the compressed data. It is often said that a picture is worth a thousand words but on the other hand, language is very powerful in capturing the essence of an image using short descriptions. With the recent success of diffusion models for text-to-image generation, we propose a generative image compression method that demonstrates the potential of saving an image as a short text embedding which in turn can be used to generate high-fidelity images which is equivalent to the original one perceptually. For a given image, its corresponding text embedding is learned using the same optimization process as the text-to-image diffusion model itself, using a learnable text embedding as input after bypassing the original transformer. The optimization is applied together with a learning compression model to achieve extreme compression of low bitrates <0.1 bpp. Based on our experiments measured by a comprehensive set of image quality metrics, our method outperforms the other state-of-the-art deep learning methods in terms of both perceptual quality and diversity.

Figures (5)

• Figure 1: Visual examples of generated images after compression with extremely low bitrates, demonstrating our method's superior capability to generate very sharp details.
• Figure 2: Overview of the sampling process of proposed generative image compression using two inputs of extremely low bitrates, where $\hat{e}_x$, a highly compressed textual embedding, is used as the conditional input for a pre-trained latent diffusion model, and $\hat{x}_g$, a highly compressed image from original image $x$, is used a constraint to guide the intermediate latent image $z^t_0$ at each time step $t$. These two are saved after the initial compression process and are the only two needed to reconstruct a high quality image $x_0$.
• Figure 3: Comparison of multiple image quality metrics for different combinations of compression guidance scale ($s_c$) and classifier-free guidance scale ($s_f$). Red in the heatmaps means higher quality while green meas lower.
• Figure 4: Visual examples of generated images after extreme compression. Our model has the average bitrate for better performance while two competitive models are subject to severe artifacts due to abnormally low bitrates, a common disadvantage of prior works where models are only trained for a target average bitrate over a large training set.
• Figure 5: Visual examples of generated images. Multiple samples from our model using the same compressed source enjoy both high perceptual quality and diversity.