Text + Sketch: Image Compression at Ultra Low Rates
Eric Lei, Yiğit Berkay Uslu, Hamed Hassani, Shirin Saeedi Bidokhti
TL;DR
This work explores ultra-low-rate image compression by repurposing pre-trained text-to-image generators to transmit concise textual prompts, optionally augmented with a spatial sketch. It introduces Prompt Inversion Compression (PIC) and its extension with spatial conditioning (PICS), achieving semantic and perceptual fidelity at rates as low as $0.002-0.013$ bpp without end-to-end training. Through evaluations on standard datasets and comparisons to HiFiC and MS-SSIM-optimized baselines, PIC and especially PICS demonstrate superior rate-perception and rate-distortion performance, highlighting the value of semantic-driven transmission and structural guidance. The findings suggest practical pathways for high-quality, ultra-efficient image compression using language and lightweight side information, with future work including human studies to assess subjective satisfaction.
Abstract
Recent advances in text-to-image generative models provide the ability to generate high-quality images from short text descriptions. These foundation models, when pre-trained on billion-scale datasets, are effective for various downstream tasks with little or no further training. A natural question to ask is how such models may be adapted for image compression. We investigate several techniques in which the pre-trained models can be directly used to implement compression schemes targeting novel low rate regimes. We show how text descriptions can be used in conjunction with side information to generate high-fidelity reconstructions that preserve both semantics and spatial structure of the original. We demonstrate that at very low bit-rates, our method can significantly improve upon learned compressors in terms of perceptual and semantic fidelity, despite no end-to-end training.