Edit-Your-Interest: Efficient Video Editing via Feature Most-Similar Propagation

TL;DR

Edit-Your-Interest introduces a lightweight zero-shot video editing framework that achieves high efficiency and fidelity by caching spatial features in a Spatio-Temporal Feature Memory and propagating the most similar cached tokens via Feature Most-Similar Propagation. An automatic cross-attention-based mask extraction and in-diffusion background injection enable precise instance editing while preserving the background. The approach demonstrates state-of-the-art performance across text alignment, temporal consistency, background preservation, and video fidelity on multiple datasets, with strong practical scalability on consumer GPUs. The work also analyzes ablations, hyperparameters, and limitations, and suggests integrating external video instance segmentation for robust multi-instance editing. Overall, the method offers a practical, scalable solution for high-quality zero-shot video editing with diffusion models.

Abstract

Text-to-image (T2I) diffusion models have recently demonstrated significant progress in video editing. However, existing video editing methods are severely limited by their high computational overhead and memory consumption. Furthermore, these approaches often sacrifice visual fidelity, leading to undesirable temporal inconsistencies and artifacts such as blurring and pronounced mosaic-like patterns. We propose Edit-Your-Interest, a lightweight, text-driven, zero-shot video editing method. Edit-Your-Interest introduces a spatio-temporal feature memory to cache features from previous frames, significantly reducing computational overhead compared to full-sequence spatio-temporal modeling approaches. Specifically, we first introduce a Spatio-Temporal Feature Memory bank (SFM), which is designed to efficiently cache and retain the crucial image tokens processed by spatial attention. Second, we propose the Feature Most-Similar Propagation (FMP) method. FMP propagates the most relevant tokens from previous frames to subsequent ones, preserving temporal consistency. Finally, we introduce an SFM update algorithm that continuously refreshes the cached features, ensuring their long-term relevance and effectiveness throughout the video sequence. Furthermore, we leverage cross-attention maps to automatically extract masks for the instances of interest. These masks are seamlessly integrated into the diffusion denoising process, enabling fine-grained control over target objects and allowing Edit-Your-Interest to perform highly accurate edits while robustly preserving the background integrity. Extensive experiments decisively demonstrate that the proposed Edit-Your-Interest outperforms state-of-the-art methods in both efficiency and visual fidelity, validating its superior effectiveness and practicality.

Figures (11)

• Figure 1: We propose Edit-Your-Interest, a zero-shot video editing method that supports both low-cost global editing (left) and precise instance local editing (right), while effectively preserving the entire background.
• Figure 2: The pipline of Edit-Your-Interest. (Top) We employ DDIM inversion to obtain the initial latent noise and then denoise the sequence via DDIM sampling. (Bottom left)We construct a Spatio-Temporal Feature Memory bank (SFM) to cache frame feature tokens, significantly reducing computational overhead. The memory is continuously updated using the SFM's update algorithm, ensuring that feature tokens remain temporally relevant throughout the video. Subsequently, Feature Most-Similar Propagation (FMP) retrieves the most similar features from the SFM and propagates them to the current frame, thereby enforcing temporal consistency in the edited video. (Bottom right) We introduce an Automatic Mask Extraction and Injection Strategy: masks for objects of interest are first extracted from cross attention maps and then seamlessly integrated into the denoising process. This in-diffusion injection effectively suppresses boundary artifacts between foreground and background regions.
• Figure 3: The visualization of SFM's update algorithm. It can store feature tokens from time $0$ to $t-1$ relatively evenly without incurring significant storage overhead.
• Figure 4: Examples of our method's results in instance object editing and global editing. As shown, our approach enables not only precise local object editing but also global editing.
• Figure 5: Comparison results between our method and direct replacement using semantic masks. It can be seen that our method mitigates boundary artifacts between foreground and background.