VIDMP3: Video Editing by Representing Motion with Pose and Position Priors

TL;DR

VidMP3 tackles motion-preserved video editing by learning a generalized motion representation from pose and position priors through the MotionGuide module. This external motion signal is injected into the temporal self-attention of a diffusion-based video editor that is initialized from a text-to-image model, enabling cross-domain and structure edits while preserving original motion. The approach achieves strong quantitative and qualitative gains over prior methods, scales to SDXL, and supports personalization and background/style edits, broadening practical video editing capabilities. By separating motion from appearance and reducing manual intervention, VidMP3 lays a foundation for flexible, temporally coherent editing across diverse content transformations.

Abstract

Motion-preserved video editing is crucial for creators, particularly in scenarios that demand flexibility in both the structure and semantics of swapped objects. Despite its potential, this area remains underexplored. Existing diffusion-based editing methods excel in structure-preserving tasks, using dense guidance signals to ensure content integrity. While some recent methods attempt to address structure-variable editing, they often suffer from issues such as temporal inconsistency, subject identity drift, and the need for human intervention. To address these challenges, we introduce VidMP3, a novel approach that leverages pose and position priors to learn a generalized motion representation from source videos. Our method enables the generation of new videos that maintain the original motion while allowing for structural and semantic flexibility. Both qualitative and quantitative evaluations demonstrate the superiority of our approach over existing methods. The code will be made publicly available at https://github.com/sandeep-sm/VidMP3.

Figures (15)

• Figure 1: VidMP3. We present a novel video editing technique that can perform challenging video editing tasks guided by pose and position priors. We introduce a MotionGuide module that learns a generalized motion representation from correspondence and depth maps. We inject the features of this module to the "Value"s of the temporal self-attention layer of a T2V initialized with a T2I model. During inference, we use the correspondence and depth maps of the source video to generate a novel motion-preserved video. VidMP3 enables the generation of challenging edits, including Ⓐ Cross-Domain editing, where objects with vastly different semantic meanings can be generated, Ⓑ Structure editing, where structure of the object can be changed significantly, and Ⓒ adaptation to various T2V editing tasks such as personalized editing.
• Figure 2: Toy experiment on learning shape-invariant motion. We trained our MotionGuide module on the original video and tested it on videos with 1) shape changes, and 2) motion changes. We show the frames for each video to the left. From the graph at the right, it may be observed that the MotionGuide module is invariant to shape change but sensitive to motion change.
• Figure 3: Comparison with prior art on motion-preserved video editing. We consider the challenging cases of Ⓐ Cross-Domain Edit -- "silver jeep" $\xrightarrow{}$ "bulldog on roller blades", and Ⓑ Structure Edit -- "monkey"$\xrightarrow{}$ "tiger". It may be observed that in the case of cross-domain editing, all baselines suffer from severe temporal inconsistencies of the subject. For the case of structure editing, Tune-A-Video produces a highly saturated video with the head pose not correctly following the pose of the input video. Similarly, FateZero also models incorrect head pose (see second row of Ⓑ). For VideoSwap we notice that the tiger has a similar humped shape like the monkey (notice the yellow circled areas), due to the keypoint correspondences being very sparse and spatially constrained signal. The sparsity of this signal results in the orientation of the face being inaccurate, resulting in a wrong head pose of the tiger in the middle row. By comparison, VidMP3 generates temporally consistent results following the input pose while making necessary changes faithful to the new concept.
• Figure 4: Visualization of correspondence and depth maps. For two frames of the video of "a dog looking out the window of a car", we show the corresponding correspondence and depth maps we obtain from off-the-shelf models. The depth segmented using the correspondence map is multiplied with the correspondence map (right-most column) and provided as input to MotionGuide.
• Figure 5: Scaling VidMP3 to SDXL. As a novel initiative, we scaled up the T2V model to utilize SDXL as the foundation model. We show that we can model more diverse concepts using this setup, owing to the stronger generation capabilities of SDXL.