Beyond Inpainting: Unleash 3D Understanding for Precise Camera-Controlled Video Generation

TL;DR

This work designs a View-Content Dual-Stream Condition mechanism that injects both the source video and the warped depth sequence rendered under the target viewpoint into the pretrained video generation model, and introduces a lightweight LoRA-based video diffusion adapter to train the framework, fully preserving the knowledge priors of VDMs.

Abstract

Camera control has been extensively studied in conditioned video generation; however, performing precisely altering the camera trajectories while faithfully preserving the video content remains a challenging task. The mainstream approach to achieving precise camera control is warping a 3D representation according to the target trajectory. However, such methods fail to fully leverage the 3D priors of video diffusion models (VDMs) and often fall into the Inpainting Trap, resulting in subject inconsistency and degraded generation quality. To address this problem, we propose DepthDirector, a video re-rendering framework with precise camera controllability. By leveraging the depth video from explicit 3D representation as camera-control guidance, our method can faithfully reproduce the dynamic scene of an input video under novel camera trajectories. Specifically, we design a View-Content Dual-Stream Condition mechanism that injects both the source video and the warped depth sequence rendered under the target viewpoint into the pretrained video generation model. This geometric guidance signal enables VDMs to comprehend camera movements and leverage their 3D understanding capabilities, thereby facilitating precise camera control and consistent content generation. Next, we introduce a lightweight LoRA-based video diffusion adapter to train our framework, fully preserving the knowledge priors of VDMs. Additionally, we construct a large-scale multi-camera synchronized dataset named MultiCam-WarpData using Unreal Engine 5, containing 8K videos across 1K dynamic scenes. Extensive experiments show that DepthDirector outperforms existing methods in both camera controllability and visual quality. Our code and dataset will be publicly available.

Figures (12)

• Figure 1: Example results synthesized by DepthDirector. DepthDirector re-shoots the source video with novel camera trajectories. By fully leveraging the 3D understanding ability of video diffusion models, we are the first framework that achieves both precise camera controllability and consistent content preservation. We visualized the novel camera trajectories alongside the video frames.
• Figure 2: Limitations of warping-based methods.Even with SOTA video depth estimators DepthCraftervideo_depth_anythingwang2025vggtwang2025pi3, reprojected pixels exhibit noisy artifacts due to inaccurate 3D geometry. It leads to unrecoverable distortion to the identity and details of the subject, especially on human faces, which is sensitive to detailed geometry.
• Figure 3: Architecture overview of DepthDirector. We render depth video and occlusion mask video under target camera viewpoint from an explicit 3D mesh, injecting it into the noise latent by projection and addition. Source video is frame-wise concatenated alongside to provide content reference. Then a LoRA-based video diffusion adapter is trained to generate video following novel camera trajectories.
• Figure 4: Illustration of our dataset: MultiCam-WarpData.
• Figure 5: Qualitative comparison with state-of-the-art methods. DepthDirector achieves both precise camera controllability and content preservation.