ObjCtrl-2.5D: Training-free Object Control with Camera Poses

TL;DR

ObjCtrl-2.5D tackles the gap in 2D-based object control for image-to-video generation by extending 2D trajectories into depth and representing motion as camera poses. The approach is training-free and leverages depth-derived 3D trajectories, triangulation to camera poses, a Layer Control Module to isolate foreground from background, and Shared Warping Latent to stabilize object motion. Key contributions include adapting global camera-motion controls to local object motion, enabling complex motions such as rotation, and achieving superior trajectory alignment versus training-free baselines while remaining competitive with training-based methods. Experiments on DAVIS-derived and synthetic ObjCtrl-Test datasets, along with user studies, show improved object-control accuracy and broader control capabilities with pleasant qualitative results.

Abstract

This study aims to achieve more precise and versatile object control in image-to-video (I2V) generation. Current methods typically represent the spatial movement of target objects with 2D trajectories, which often fail to capture user intention and frequently produce unnatural results. To enhance control, we present ObjCtrl-2.5D, a training-free object control approach that uses a 3D trajectory, extended from a 2D trajectory with depth information, as a control signal. By modeling object movement as camera movement, ObjCtrl-2.5D represents the 3D trajectory as a sequence of camera poses, enabling object motion control using an existing camera motion control I2V generation model (CMC-I2V) without training. To adapt the CMC-I2V model originally designed for global motion control to handle local object motion, we introduce a module to isolate the target object from the background, enabling independent local control. In addition, we devise an effective way to achieve more accurate object control by sharing low-frequency warped latent within the object's region across frames. Extensive experiments demonstrate that ObjCtrl-2.5D significantly improves object control accuracy compared to training-free methods and offers more diverse control capabilities than training-based approaches using 2D trajectories, enabling complex effects like object rotation. Code and results are available at https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/.

Figures (17)

• Figure 1: Control Results of ObjCtrl-2.5D. ObjCtrl-2.5D enables versatile object motion control for image-to-video generation. It accepts 2D trajectories (transformed to 3D), or camera poses as control guidance (all transformed to camera poses) and achieves precise motion control by utilizing an existing camera motion control module without additional training. Unlike existing methods based on 2D trajectories, ObjCtrl-2.5D supports complex motion control beyond planar movement, such as object rotation in the last row. We strongly encourage consulting our https://wzhouxiff.github.io/projects/ObjCtrl-2.5D/ for dynamic results, as they cannot be effectively represented through still images.
• Figure 2: Object control results using 2D and 3D trajectories. On the left, the red line represents the 2D trajectory, the blue line indicates the 3D trajectory extracted from real-world video in DAVIS davis, and the green point marks the starting point of the trajectory. The training-based method DragAnything draganything, which controls objects using a 2D trajectory, closely follows the specified path; however, it results in the car appearing to move horizontally toward the grass, which is atypical in real-world settings. By incorporating depth information from a 3D trajectory, our proposed method generates videos that not only follow the spatial trajectory but also achieve more realistic movement.
• Figure 3: Framework of ObjCtrl-2.5D. ObjCtrl-2.5D first extends the provided 2D trajectory $\mathcal{T}_{2d}$ to a 3D trajectory $\mathcal{T}_{3d}$ using depth information from the conditioning image. This 3D trajectory is then transformed into a camera pose $\mathbf{E_o}$ via triangulation hartley2003multiple. To achieve object motion control within a frozen camera motion control module, ObjCtrl-2.5D integrates a Layer Control Module (LCM) that separates the object and background with distinct camera poses ($\mathbf{E_o}$ and $\mathbf{E_{bg}}$). After extracting camera pose features via a Camera Encoder, LCM spatially combines these features using a series of scale-wise masks. Additionally, ObjCtrl-2.5D introduces a Shared Warping Latent (SWL) technique, implemented with a 3D low-pass filter $H$, to enhance control by sharing low-frequency initialized noise across frames within the warped areas of the object.
• Figure 4: 3D Trajectory to Camera Poses. We model the object movement in a video, indicated by a 3D trajectory, as the camera's location translation in 3D space. Details refer to Sec. \ref{['sec:3d_to_cam']} and Algorithm. \ref{['alg:3d_to_cam']}.
• Figure 5: Qualitative Comparison with Training-free Methods. Compared to PEEKABOO peekaboo and FreeTraj freetraj that coarsely move the objects within the bounding boxes derived from the trajectory, our ObjCtrl-2.5D achieves higher trajectory alignment by extending the 2D trajectory to 3D and accurately transforming it into camera poses through triangulation hartley2003multiple.