ConsistEdit: Highly Consistent and Precise Training-free Visual Editing

TL;DR

This work tackles the problem of achieving strong, consistent, training-free text-guided visual editing in MM-DiT-based diffusion models, addressing challenges in multi-round and video editing where errors accumulate. By analyzing MM-DiT's attention and identifying three key insights, the authors propose ConsistEdit, a vision-only attention-control method that uses mask-guided pre-attention fusion and differentiated manipulation of $Q$, $K$, and $V$ tokens to deliver prompt-aligned edits while preserving non-edited content. The approach demonstrates state-of-the-art performance across image and video editing tasks, supporting both structure-consistent and structure-inconsistent edits, multi-region editing, and progressive control of structural consistency via a parameter $oldsymbol{\alpha}$. It also generalizes across MM-DiT variants and video models, enabling reliable, real-time editing without handcraft, and enabling robust multi-round editing and fine-grained control with practical impact for content creation. Overall, ConsistEdit leverages architecture-aware attention control to unlock reliable, flexible editing in MM-DiT foundations, with broad applicability to image and video domains.

Abstract

Recent advances in training-free attention control methods have enabled flexible and efficient text-guided editing capabilities for existing generation models. However, current approaches struggle to simultaneously deliver strong editing strength while preserving consistency with the source. This limitation becomes particularly critical in multi-round and video editing, where visual errors can accumulate over time. Moreover, most existing methods enforce global consistency, which limits their ability to modify individual attributes such as texture while preserving others, thereby hindering fine-grained editing. Recently, the architectural shift from U-Net to MM-DiT has brought significant improvements in generative performance and introduced a novel mechanism for integrating text and vision modalities. These advancements pave the way for overcoming challenges that previous methods failed to resolve. Through an in-depth analysis of MM-DiT, we identify three key insights into its attention mechanisms. Building on these, we propose ConsistEdit, a novel attention control method specifically tailored for MM-DiT. ConsistEdit incorporates vision-only attention control, mask-guided pre-attention fusion, and differentiated manipulation of the query, key, and value tokens to produce consistent, prompt-aligned edits. Extensive experiments demonstrate that ConsistEdit achieves state-of-the-art performance across a wide range of image and video editing tasks, including both structure-consistent and structure-inconsistent scenarios. Unlike prior methods, it is the first approach to perform editing across all inference steps and attention layers without handcraft, significantly enhancing reliability and consistency, which enables robust multi-round and multi-region editing. Furthermore, it supports progressive adjustment of structural consistency, enabling finer control.

Figures (21)

• Figure 1: Visualization of projected Q, K, V vision tokens in attention layers of the MM-DiT blocks at 15th sampling step of prompt "A standing horse."
• Figure 2: (a) shows the ConsistEdit pipeline. Given a real image or video $\bm{I}_s$ and source text tokens $\bm{\mathrm{P}}_s$, we first invert the source to obtain the vision tokens $\bm{z}^T$, which is concatenated with the target prompt tokens $\bm{\mathrm{P}}_{tg}$ and passed into the generation process to produce the edited image or video $\bm{I}_{tg}$. During inference, a mask $\bm{M}$ generated by our extraction method delineates editing and non-editing regions. We apply structure and content fusion to enable prompt-aligned edits while preserving structural consistency within edited regions and maintaining content integrity elsewhere. (b) illustrates the mask-guided attention fusion for timesteps where $t>(1-\alpha)T$, which is applied exclusively to the vision parts, while the text parts remain unchanged.
• Figure 3: Comparison of V token swapping strategies for content consistency. Swapping vision-only V tokens leads to superior content consistency under high consistency strength settings, while maintaining comparable editing capability to original methods when the consistency strength is low.
• Figure 4: Real image multi-round editing results. Starting from a real image, we first perform inversion to project it into the latent space. We then sequentially edit the clothing color, motion, and hair.
• Figure 5: Qualitative comparison of methods on real image editing tasks.