Edit as You See: Image-guided Video Editing via Masked Motion Modeling

TL;DR

This work introduces IVEDiff, the first diffusion-based framework for image-guided video editing that edits regions in a video by referencing a target image and masks rather than relying on text prompts. It combines an optical-flow-guided motion reference network (MotRefNet) with a masked motion modeling (MMM) fine-tuning strategy to preserve temporal coherence while maintaining high per-frame edit quality, built atop MimicBrush and augmented with CLIP guidance. A new IVE-Benchmark is proposed to evaluate inter-frame consistency and frame quality across texture transfer and object modification tasks. Experimental results show IVEDiff achieves temporally smooth edited videos and robust performance across editing targets, highlighting the value of motion priors and self-supervised fine-tuning in image-guided video editing.

Abstract

Recent advancements in diffusion models have significantly facilitated text-guided video editing. However, there is a relative scarcity of research on image-guided video editing, a method that empowers users to edit videos by merely indicating a target object in the initial frame and providing an RGB image as reference, without relying on the text prompts. In this paper, we propose a novel Image-guided Video Editing Diffusion model, termed IVEDiff for the image-guided video editing. IVEDiff is built on top of image editing models, and is equipped with learnable motion modules to maintain the temporal consistency of edited video. Inspired by self-supervised learning concepts, we introduce a masked motion modeling fine-tuning strategy that empowers the motion module's capabilities for capturing inter-frame motion dynamics, while preserving the capabilities for intra-frame semantic correlations modeling of the base image editing model. Moreover, an optical-flow-guided motion reference network is proposed to ensure the accurate propagation of information between edited video frames, alleviating the misleading effects of invalid information. We also construct a benchmark to facilitate further research. The comprehensive experiments demonstrate that our method is able to generate temporally smooth edited videos while robustly dealing with various editing objects with high quality.

Figures (9)

• Figure 1: Based on the given reference image and masks indicating the regions to be edited in the source video, our IVEDiff can automatically extract semantically related information from the reference image to edit the video without the need for additional text descriptions, while ensuring the temporal smoothness. [Best viewed with zoom-in.]
• Figure 2: Overall pipeline of the proposed masked motion modeling fine-tuning strategy. To align the fine-tuning process with the inference process, we use the first frame of the video clip in the training set as the reference image, and partially obscure the remaining frames. By this fine-tuning process, model gains the capabilities of capturing inter-frame temporal consistency while preserving the base image model's original ability to model intra-frame semantic correlations.
• Figure 3: Overall framework of the proposed IVEDiff.
• Figure 4: The qualitative comparisons between our methods with baselines. For the texture transfer, we use the fine-grained masks which are generated by SAM2 to indicate the editing target, and use the highly semantic-related reference image; For the object modification, we use the rectangle masks which coarsely cover the editing target and the semantically unrelated reference image.
• Figure 5: Qualitative comparison of main components.