Stable Diffusion is a Natural Cross-Modal Decoder for Layered AI-generated Image Compression

TL;DR

The paper tackles compressing AI-generated images (AIGIs) at ultra-low bitrates by introducing a layered cross-modal compression framework that partitions information into semantic, structure, and texture priors. It uses Stable Diffusion as a natural cross-modal decoder, leveraging text prompts, edge/pose maps, and color palettes to reconstruct images progressively and enables editing directly on the bitstream. Key contributions include a practical encoding scheme for the three priors, integration of a T2I-Adapter with SD, and empirical results showing superior perceptual quality at <0.02 bpp compared with traditional codecs, plus editable bitstreams without full decoding. This approach opens new directions for scalable, controllable AIGI compression and editing workflows.

Abstract

Recent advances in Artificial Intelligence Generated Content (AIGC) have garnered significant interest, accompanied by an increasing need to transmit and compress the vast number of AI-generated images (AIGIs). However, there is a noticeable deficiency in research focused on compression methods for AIGIs. To address this critical gap, we introduce a scalable cross-modal compression framework that incorporates multiple human-comprehensible modalities, designed to efficiently capture and relay essential visual information for AIGIs. In particular, our framework encodes images into a layered bitstream consisting of a semantic layer that delivers high-level semantic information through text prompts; a structural layer that captures spatial details using edge or skeleton maps; and a texture layer that preserves local textures via a colormap. Utilizing Stable Diffusion as the backend, the framework effectively leverages these multimodal priors for image generation, effectively functioning as a decoder when these priors are encoded. Qualitative and quantitative results show that our method proficiently restores both semantic and visual details, competing against baseline approaches at extremely low bitrates ( <0.02 bpp). Additionally, our framework facilitates downstream editing applications without requiring full decoding, thereby paving a new direction for future research in AIGI compression.

Figures (5)

• Figure 1: Overview of the proposed layered AGI compression framework using Stable Diffusion as a cross-modal decoder.
• Figure 2: The qualitative comparison results of JPEG2000, VVC, Perco and our method using poses as structure information on the AGIQA dataset.
• Figure 3: The qualitative comparison results of JPEG2000, VVC, Perco and our method using edges as structure information on the AGIQA dataset.
• Figure 4: The R-D performance of JPEG2000, VVC, Perco and our method on the AGIQA dataset. Layer1, Layer2 and Layer3 denotes images incrementally reconstructed from semantic, structure and texturte priors, respectively.
• Figure 5: Results of Structure Manipulation, Texture Synthesis and Object Erasing. The first column shows the unchanged texture/structure map, the remaining columns shows the editing results. The edited structure/texture map is plotted in the corner of each editing result respectively.