Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models

TL;DR

The paper tackles text-driven image editing with diffusion models, focusing on the inversion bottleneck needed to edit real images while preserving fidelity. It introduces Prompt Tuning Inversion (PTI), a two-stage approach that first encodes input content into a learnable conditional embedding during reconstruction and then edits by linearly interpolating between this optimized embedding and the target text embedding under classifier-free guidance. PTI achieves a better trade-off between editability and fidelity than state-of-the-art baselines, demonstrated on ImageNet using Latent Diffusion Models like Stable Diffusion. The method is user-friendly (no masks or source-image descriptions required), computationally efficient, and generalizable across domains, with potential for further refinement in multi-object editing through more precise attention control.

Abstract

Recently large-scale language-image models (e.g., text-guided diffusion models) have considerably improved the image generation capabilities to generate photorealistic images in various domains. Based on this success, current image editing methods use texts to achieve intuitive and versatile modification of images. To edit a real image using diffusion models, one must first invert the image to a noisy latent from which an edited image is sampled with a target text prompt. However, most methods lack one of the following: user-friendliness (e.g., additional masks or precise descriptions of the input image are required), generalization to larger domains, or high fidelity to the input image. In this paper, we design an accurate and quick inversion technique, Prompt Tuning Inversion, for text-driven image editing. Specifically, our proposed editing method consists of a reconstruction stage and an editing stage. In the first stage, we encode the information of the input image into a learnable conditional embedding via Prompt Tuning Inversion. In the second stage, we apply classifier-free guidance to sample the edited image, where the conditional embedding is calculated by linearly interpolating between the target embedding and the optimized one obtained in the first stage. This technique ensures a superior trade-off between editability and high fidelity to the input image of our method. For example, we can change the color of a specific object while preserving its original shape and background under the guidance of only a target text prompt. Extensive experiments on ImageNet demonstrate the superior editing performance of our method compared to the state-of-the-art baselines.

Figures (7)

• Figure 1: Illustration of different methods in editing the color of the car. Methods based on the original DDIM inversion (i.e., DDIM-Edit, DiffEdit and DiffEdit w/o src) cannot preserve the shape of the car. In contrast, our method successfully changes the color while preserving the structural information. The target text is "a yellow car". The source text is "a red car" for DiffEdit.
• Figure 2: An overview of our proposed image editing method. Stage 1: we first apply DDIM inversion to the input image embedding to obtain a diffusion trajectory $\{z_t^*\}_{t=0}^T$. Then we reconstruct the input image along with the reversed trajectory by optimizing the learnable conditional embedding $c_t$. Stage 2: we perform classifier-free guidance sampling conditioned on a linear interpolation between target embedding $c^*$ and $c_t$ at each diffusion step. $\bigoplus$ denotes element-wise weighted addition. Red dashes indicate the gradient flow in PTI.
• Figure 3: Reconstruction quality of our Prompt Tuning Inversion, and DDIM inversion with different classifier-free guidance (CFG) scales $\omega$ in the sampling process. $\omega\!=\!0$ in the forward process for all methods.
• Figure 4: Comparison with DiffEdit and DDIM-Edit on ImageNet. For all methods, we set the DDIM encoding ratio to 0.8, and only vary the mask threshold to draw the trade-off curve.
• Figure 5: Editing examples on ImageNet by our method and other methods. DDIM-G/DDIM-L indicates the baseline method DDIM-Edit with/without the local blend trick.