MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance

TL;DR

MagicMotion introduces a trajectory-controlled image-to-video framework that supports dense (masks) to sparse (boxes, sparse boxes) trajectory signals via a Trajectory ControlNet embedded in a diffusion-transformer backbone. It implements a three-stage progressive training and a Latent Segmentation Loss to preserve fine-grained object shapes under sparse cues, and pairs this with MagicData, a large annotated 23K-video dataset, and MagicBench, a 600-video benchmark across varying object counts. The approach achieves superior video quality and trajectory adherence compared with prior methods, validated on MagicBench and DAVIS, with ablations confirming the value of the dataset, progressive training, and latent segmentation losses. These contributions provide a public dataset and benchmark for trajectory-controlled video generation and demonstrate practical improvements in multi-object, long-sequence video synthesis.

Abstract

Recent advances in video generation have led to remarkable improvements in visual quality and temporal coherence. Upon this, trajectory-controllable video generation has emerged to enable precise object motion control through explicitly defined spatial paths. However, existing methods struggle with complex object movements and multi-object motion control, resulting in imprecise trajectory adherence, poor object consistency, and compromised visual quality. Furthermore, these methods only support trajectory control in a single format, limiting their applicability in diverse scenarios. Additionally, there is no publicly available dataset or benchmark specifically tailored for trajectory-controllable video generation, hindering robust training and systematic evaluation. To address these challenges, we introduce MagicMotion, a novel image-to-video generation framework that enables trajectory control through three levels of conditions from dense to sparse: masks, bounding boxes, and sparse boxes. Given an input image and trajectories, MagicMotion seamlessly animates objects along defined trajectories while maintaining object consistency and visual quality. Furthermore, we present MagicData, a large-scale trajectory-controlled video dataset, along with an automated pipeline for annotation and filtering. We also introduce MagicBench, a comprehensive benchmark that assesses both video quality and trajectory control accuracy across different numbers of objects. Extensive experiments demonstrate that MagicMotion outperforms previous methods across various metrics. Our project page are publicly available at https://quanhaol.github.io/magicmotion-site.

Figures (20)

• Figure 1: Example videos generated by MagicMotion. MagicMotion consists of three stages, each supporting a different level of control from dense to sparse: mask, box, and sparse box. Given an input image and any form of trajectory, MagicMotion can generate high-quality videos, animating objects in the image to move along the user-specified path.
• Figure 2: Overview of MagicMotion Architecture (text prompt and encoder are omitted for simplicity). MagicMotion employs a pretrained 3D VAE to encode the input trajectory, first-frame image, and training video into latent space. It has two separate branches: the video branch processes video and image tokens, and the trajectory branch uses Trajectory ControlNet to fuse trajectory and image tokens, which is later integrated to the video branch through a zero-initialized convolution layer. Besides, diffusion features from DiT blocks are concatenated and processed by a trainable segment head to predict latent segmentation masks, which contribute to our latent segment loss.
• Figure 3: Overview of the Dataset Pipeline. The Curation Pipeline is used to construct trajectory annotations, while the Filtering Pipeline filters out unsuitable videos for training.
• Figure 4: Comparison results of different object number on MagicBench. To present the results more clearly, we have negated the FVD and FID scores.
• Figure 5: MagicMotion successfully controls the main objects moving along the provided trajectory, while all other methods exhibit significant defects marked with the orange box.