ProEdit: Inversion-based Editing From Prompts Done Right

TL;DR

ProEdit tackles the pervasive issue of excessive source-image information intrusion in inversion-based editing by addressing both attention and latent distributions. It introduces KV-mix to regionally mix source and target attention features and Latents-Shift to AdaIN-style shift the inverted latent in edited regions, both without retraining. The method is plug-and-play and yields state-of-the-art results on image and video editing benchmarks, while preserving non-edited content and background structure. This approach enables more accurate attribute edits guided by prompts and demonstrates strong practical impact for flow-based editing pipelines.

Abstract

Inversion-based visual editing provides an effective and training-free way to edit an image or a video based on user instructions. Existing methods typically inject source image information during the sampling process to maintain editing consistency. However, this sampling strategy overly relies on source information, which negatively affects the edits in the target image (e.g., failing to change the subject's atributes like pose, number, or color as instructed). In this work, we propose ProEdit to address this issue both in the attention and the latent aspects. In the attention aspect, we introduce KV-mix, which mixes KV features of the source and the target in the edited region, mitigating the influence of the source image on the editing region while maintaining background consistency. In the latent aspect, we propose Latents-Shift, which perturbs the edited region of the source latent, eliminating the influence of the inverted latent on the sampling. Extensive experiments on several image and video editing benchmarks demonstrate that our method achieves SOTA performance. In addition, our design is plug-and-play, which can be seamlessly integrated into existing inversion and editing methods, such as RF-Solver, FireFlow and UniEdit.

Figures (11)

• Figure 1: ProEdit for image and video editing. We propose a highly accurate, plug-and-play editing method for flow inversion that addresses the problem of excessive source image information injection, which prevents proper modification of attributes such as pose, number, and color. Our method has demonstrated impressive performance in both image editing and video editing tasks.
• Figure 2: Framework comparison between (a) previous methods and (b) our method. To address the issue of excessive source image information injection, we introduce the Shift module for inverted noise and the Mix module for the attention injection, alleviating the editing failures caused by these issues.
• Figure 3: Excessive source image information injection phenomenon in RF-Solver. We validate it by visualizing the attention from source and target text tokens to the visual tokens during initial and sampling stage. In RF-Solver, the attention from the source text token to the visual tokens remains higher than that from the target text token. However, after removing attention injection, the attention from "black" and "orange" to visual tokens returns to similar levels, but some subject attributes (e.g., pose) change accordingly.
• Figure 4: Pipeline of our ProEdit. The mask extraction module identifies the edited region based on source and target prompts during the first inversion step. After obtaining the inverted noise, we apply Latents-Shift to perturb the initial distribution in the edited region, reducing source image information. In selected sampling steps, we fuse source and target attention features in the edited region while directly injecting source features in non-edited regions to achieve accurate attribute editing and background preservation simultaneously.
• Figure 5: Qualitative comparison on image editing. With our method, various flow-based inversion methods achieve more appropriate editing while preserving the consistency of background and non-editing content.