LeviTor: 3D Trajectory Oriented Image-to-Video Synthesis

TL;DR

LeviTor tackles the ambiguity of 2D trajectory control in image-to-video synthesis by introducing depth-aware, clustered-point control for 3D trajectories. It represents object motion with depth-augmented K-means points derived from mask maps and trains a diffusion-based video generator (Stable Video Diffusion with ControlNet) to follow these 3D trajectories. The approach leverages SAM2/SA-V for rich mask data and DepthAnythingV2 for relative depth, enabling interactive 3D trajectory input via a user-friendly 2D interface. Quantitative and qualitative results show LeviTor outperforms 2D-only baselines (e.g., DragAnything, DragNUWA) in FVD/FID and occlusion/depth handling, while ablations verify the contribution of depth and instance information and the trade-off with control-point count. Limitations include dependence on segmentation accuracy, lack of true 3D physics modeling, and reliance on the base video model; future work aims to handle non-rigid motion with stronger video backbones and tracking data.

Abstract

The intuitive nature of drag-based interaction has led to its growing adoption for controlling object trajectories in image-to-video synthesis. Still, existing methods that perform dragging in the 2D space usually face ambiguity when handling out-of-plane movements. In this work, we augment the interaction with a new dimension, i.e., the depth dimension, such that users are allowed to assign a relative depth for each point on the trajectory. That way, our new interaction paradigm not only inherits the convenience from 2D dragging, but facilitates trajectory control in the 3D space, broadening the scope of creativity. We propose a pioneering method for 3D trajectory control in image-to-video synthesis by abstracting object masks into a few cluster points. These points, accompanied by the depth information and the instance information, are finally fed into a video diffusion model as the control signal. Extensive experiments validate the effectiveness of our approach, dubbed LeviTor, in precisely manipulating the object movements when producing photo-realistic videos from static images. Our code is available at: https://github.com/ant-research/LeviTor.

Figures (11)

• Figure 1: An example of object movement and occlusion represented by K-means clustered points.
• Figure 2: Control signal generation process of LeviTor.
• Figure 3: Inference pipeline of LeviTor, which consists of user retrieval panel, interactive panel, 3D rendered object masks generation and video synthesis. Users can easily draw 3D trajectories through our retrieval panel and interactive panel, and our system later use these inputs to generate user desired videos.
• Figure 4: 3D rendered object masks generation pipeline.
• Figure 5: Qualitative comparison with DragAnything DragAnything and DragNUWA DragNUWA. LeviTor and DragAnything both support moving user-selected mask areas, whereas DragNUWA directly encodes trajectories as control signals and does not support user selection of operation areas. The top two rows show evaluation on control of mutual occlusion between objects. The left bottom images show comparison of forward and backward object movements control. The right bottom images show a case of complex motion implementation.