Motion Control for Enhanced Complex Action Video Generation

TL;DR

A novel framework, MVideo, designed to produce long-duration videos with precise, fluid actions, overcomes the limitations of text prompts by incorporating mask sequences as an additional motion condition input, providing a clearer, more accurate representation of intended actions.

Abstract

Existing text-to-video (T2V) models often struggle with generating videos with sufficiently pronounced or complex actions. A key limitation lies in the text prompt's inability to precisely convey intricate motion details. To address this, we propose a novel framework, MVideo, designed to produce long-duration videos with precise, fluid actions. MVideo overcomes the limitations of text prompts by incorporating mask sequences as an additional motion condition input, providing a clearer, more accurate representation of intended actions. Leveraging foundational vision models such as GroundingDINO and SAM2, MVideo automatically generates mask sequences, enhancing both efficiency and robustness. Our results demonstrate that, after training, MVideo effectively aligns text prompts with motion conditions to produce videos that simultaneously meet both criteria. This dual control mechanism allows for more dynamic video generation by enabling alterations to either the text prompt or motion condition independently, or both in tandem. Furthermore, MVideo supports motion condition editing and composition, facilitating the generation of videos with more complex actions. MVideo thus advances T2V motion generation, setting a strong benchmark for improved action depiction in current video diffusion models. Our project page is available at https://mvideo-v1.github.io/.

Figures (9)

• Figure 1: Complex action videos generated by MVideo, with a duration of 12 seconds and a spatial resolution of 480 $\times$ 720 pixels.
• Figure 2: Inference Pipeline: Given a text prompt and a simple description identifying the reference object in the video, MVideo automatically extracts the corresponding mask sequence and iteratively generates long-duration motion videos, using the mask sequence as an additional condition.
• Figure 3: Training Pipeline: MVideo takes a text prompt as input, along with a mask sequence, a high-resolution image, and a low-resolution video as conditions. During training, diffusion loss adapts these new conditions, while consistency loss preserves the model's text-alignment capability.
• Figure 4: Visual comparison of videos generated by OpenSora-v1.2 opensora, CogVideoX-5b yang2024cogvideox, and MVideo-5b.
• Figure 5: MVideo generates videos with identical motion conditions and varied text prompts to change scenes. Each video is 12 seconds long with a resolution of 480 $\times$ 720 pixels.