DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing

TL;DR

DeltaSpace identifies a semantic-aligned CLIP delta space where image and text feature differences align, enabling text-free training for flexible text-guided image editing. DeltaEdit learns a coarse-to-fine mapping from CLIP image differences to StyleGAN's latent-space directions and uses CLIP text differences at inference to drive edits, allowing zero-shot applicability to unseen prompts. The framework is instantiated in both GAN (DeltaEdit-G) and diffusion (DeltaEdit-D) backbones, showing superior editing quality, disentanglement, and reconstruction across multiple datasets. Extensive experiments demonstrate improved efficiency, generalization to unseen prompts, and robust editing performance without per-prompt tuning, highlighting practical impact for controllable image editing with CLIP priors.

Abstract

Text-guided image editing faces significant challenges when considering training and inference flexibility. Much literature collects large amounts of annotated image-text pairs to train text-conditioned generative models from scratch, which is expensive and not efficient. After that, some approaches that leverage pre-trained vision-language models have been proposed to avoid data collection, but they are limited by either per text-prompt optimization or inference-time hyper-parameters tuning. To address these issues, we investigate and identify a specific space, referred to as CLIP DeltaSpace, where the CLIP visual feature difference of two images is semantically aligned with the CLIP textual feature difference of their corresponding text descriptions. Based on DeltaSpace, we propose a novel framework called DeltaEdit, which maps the CLIP visual feature differences to the latent space directions of a generative model during the training phase, and predicts the latent space directions from the CLIP textual feature differences during the inference phase. And this design endows DeltaEdit with two advantages: (1) text-free training; (2) generalization to various text prompts for zero-shot inference. Extensive experiments validate the effectiveness and versatility of DeltaEdit with different generative models, including both the GAN model and the diffusion model, in achieving flexible text-guided image editing. Code is available at https://github.com/Yueming6568/DeltaEdit.

Figures (18)

• Figure 1: Feature space analysis on multimodal datasets including (a) MultiModal-CelebA-HQ xia2021tedigan dataset and (b) MS-COCO lin2014microsoftchen2015microsoft dataset. Paired CLIP image-text features (marked in red and green) and paired CLIP delta image-text features (marked in orange and brown) are visualized in 2D using t-SNE visualization.
• Figure 2: Examples of text-guided image editing enabled by our DeltaEdit applied to the GAN model (named DeltaEdit-G) and the diffusion model (named DeltaEdit-D), respectively.
• Figure 3: Illustration of the straightforward solution to text-free training.
• Figure 4: The editing results of the straightforward solution. Take different source images as input, the method fits them all to an average face corresponding to the text-related attributes, by directly replacing the condition from image feature $i$ to text feature $t$.
• Figure 5: The overall framework of the proposed DeltaEdit. (a) In the text-free training phase, we extract embeddings of two randomly selected images on CLIP image space and StyleGAN $\mathcal{S}$ space. Then we feed $i_1$, $s_1$ and $\Delta i$ into Delta Mapper to predict editing direction $\Delta \hat{s}$, which is supervised by $\Delta s$. (b) The detailed architecture of the Delta Mapper, which achieves coarse-to-fine editing in three levels. (c) In the inference phase, based on the co-linearity between $\Delta i$ and $\Delta t$ in CLIP joint space, DeltaEdit can achieve text-guided image editing by taking two text prompts (denoting the source and desired target) as input.