Uni3C: Unifying Precisely 3D-Enhanced Camera and Human Motion Controls for Video Generation

TL;DR

Uni3C tackles the challenge of controllable video generation by unifying precise 3D-guided camera and human motion controls. It introduces a lightweight PCDController that leverages monocular-depth–derived point clouds and 3D priors, and a global 3D world guidance that aligns scene geometry and SMPL-X human signals in a common environment during inference. The approach enables training in separate domains (camera vs. human motion) without joint annotated data while delivering coherent camera-humanoid motion in 3D space, validated by comprehensive camera-control benchmarks and unified camera–human motion tests. The results demonstrate superior controllability and motion quality over existing methods, with robust generalization to out-of-distribution content and motion transfer scenarios, highlighting practical impact for flexible, 3D-aware video generation.

Abstract

Camera and human motion controls have been extensively studied for video generation, but existing approaches typically address them separately, suffering from limited data with high-quality annotations for both aspects. To overcome this, we present Uni3C, a unified 3D-enhanced framework for precise control of both camera and human motion in video generation. Uni3C includes two key contributions. First, we propose a plug-and-play control module trained with a frozen video generative backbone, PCDController, which utilizes unprojected point clouds from monocular depth to achieve accurate camera control. By leveraging the strong 3D priors of point clouds and the powerful capacities of video foundational models, PCDController shows impressive generalization, performing well regardless of whether the inference backbone is frozen or fine-tuned. This flexibility enables different modules of Uni3C to be trained in specific domains, i.e., either camera control or human motion control, reducing the dependency on jointly annotated data. Second, we propose a jointly aligned 3D world guidance for the inference phase that seamlessly integrates both scenic point clouds and SMPL-X characters to unify the control signals for camera and human motion, respectively. Extensive experiments confirm that PCDController enjoys strong robustness in driving camera motion for fine-tuned backbones of video generation. Uni3C substantially outperforms competitors in both camera controllability and human motion quality. Additionally, we collect tailored validation sets featuring challenging camera movements and human actions to validate the effectiveness of our method.

Figures (19)

• Figure 1: The overview of Uni3C, which adopts multi-modal conditions. The camera, point clouds, and reference image are assigned to the camera control module called PCDController, while the reference image, SMPL-X pavlakos2019expressive, and Hamer hamer are assigned to human animation called RealisDance-DiT zhou2025RealisDance.
• Figure 2: Pipeline of PCDController, which is built as a lightweight DiT trained from scratch. We first obtain point clouds via monocular depth from the first view. Then, the point clouds are warped and rendered into the video $V_{pcd}$. Input conditions for PCDController comprise rendered $V_{pcd}$, Plücker ray $\mathbf{P}$, and the noisy latent $z_t$. Only the PCDController and camera encoder are trainable in our framework. For inference of unified control over camera and human motions, we directly replace the frozen Wan backbone with RealisDance-DiT zhou2025RealisDance without joint fine-tuning.
• Figure 3: Results of PCDController with imperfect point clouds. Benefiting from the well-preserved capacity of VDM, PCDController enjoys robust generation with inferior point clouds.
• Figure 4: Overview of global 3D world guidance. (a) We first align SMPL-X characters from the human world space $W_{hum}$ to the environment world space $W_{env}$ with dense point clouds. (b) GeoCalib veicht2024geocalib is used to calibrate the gravity direction of SMPL-X. (c) Rigid transformation coefficients $\tilde{s},\tilde{R},\tilde{t}$ are employed to align the whole SMPL-X sequence. We re-render all aligned conditions under specific camera trajectories as the global 3D world guidance.
• Figure 5: Results of unified camera and human motion controls. Leftmost images are reference views; the first row indicates aligned 3D world guidance.