Consistent Video Editing as Flow-Driven Image-to-Video Generation

TL;DR

FlowV2V reinterprets video editing as flow-driven image-to-video generation to robustly handle non-rigid motion while maintaining temporal coherence. It introduces a four-part pipeline—First Frame Editing, Iterative Motion Propagation, Shape-Consistent Flow Calibration, and Flow-Driven I2V Generation—together with shape-aligned flow deformation and a flow-calibration network to produce reliable motion guidance. Evaluations on DAVIS-EDIT demonstrate superior sample quality and temporal consistency, with ablations confirming the benefits of optical-flow conditioning over depth-based controls. The approach broadens editable scenarios to multi-object and portrait editing, reducing artifacts in non-rigid transformations and offering practical improvements for video editing workflows.

Abstract

With the prosper of video diffusion models, down-stream applications like video editing have been significantly promoted without consuming much computational cost. One particular challenge in this task lies at the motion transfer process from the source video to the edited one, where it requires the consideration of the shape deformation in between, meanwhile maintaining the temporal consistency in the generated video sequence. However, existing methods fail to model complicated motion patterns for video editing, and are fundamentally limited to object replacement, where tasks with non-rigid object motions like multi-object and portrait editing are largely neglected. In this paper, we observe that optical flows offer a promising alternative in complex motion modeling, and present FlowV2V to re-investigate video editing as a task of flow-driven Image-to-Video (I2V) generation. Specifically, FlowV2V decomposes the entire pipeline into first-frame editing and conditional I2V generation, and simulates pseudo flow sequence that aligns with the deformed shape, thus ensuring the consistency during editing. Experimental results on DAVIS-EDIT with improvements of 13.67% and 50.66% on DOVER and warping error illustrate the superior temporal consistency and sample quality of FlowV2V compared to existing state-of-the-art ones. Furthermore, we conduct comprehensive ablation studies to analyze the internal functionalities of the first-frame paradigm and flow alignment in the proposed method.

Figures (12)

• Figure 1: Qualitative comparisons of non-rigid motion editing (top), including face rotation, and additional applications (bottom), such as face editing and multi-motion, using our proposed FlowV2V.
• Figure 2: The overall architecture of our proposed FlowV2V video editing pipeline. It consists of a first-frame editor, an iterative motion propagator, a shape-consistent flow calibration module, a reference encoder capturing multi-scale features from the source image, and a trainable SVD encoder initialized from SVD, serving as the final spatial-temporal feature merger in generation guidance.
• Figure 3: Qualitative comparison of FlowV2V with state-of-the-art methods.
• Figure 4: Human portrait and multi-object editing results with FlowV2V.
• Figure 5: Qualitative results for the ablation study of different first-frame inputs.