DualCamCtrl: Dual-Branch Diffusion Model for Geometry-Aware Camera-Controlled Video Generation

TL;DR

DualCamCtrl tackles geometry-aware camera-controlled video generation by introducing a dual-branch diffusion framework that simultaneously models RGB and depth conditioned on shared camera poses. The SematIc Guided Mutual Alignment (SIGMA) and a 3D fusion block enable robust RGB-D interaction, while a two-stage training regime stabilizes learning and improves cross-modal consistency. Empirical results on RealEstate10K and DL3DV show over 40% reductions in rotation errors and superior metrics/qualitative judgments compared to state-of-the-art baselines. The work provides insights into the distinct roles of depth and early/late denoising stages, offering a practical path toward more faithful camera-driven video synthesis and broader implications for geometry-aware diffusion models.

Abstract

This paper presents DualCamCtrl, a novel end-to-end diffusion model for camera-controlled video generation. Recent works have advanced this field by representing camera poses as ray-based conditions, yet they often lack sufficient scene understanding and geometric awareness. DualCamCtrl specifically targets this limitation by introducing a dual-branch framework that mutually generates camera-consistent RGB and depth sequences. To harmonize these two modalities, we further propose the Semantic Guided Mutual Alignment (SIGMA) mechanism, which performs RGB-depth fusion in a semantics-guided and mutually reinforced manner. These designs collectively enable DualCamCtrl to better disentangle appearance and geometry modeling, generating videos that more faithfully adhere to the specified camera trajectories. Additionally, we analyze and reveal the distinct influence of depth and camera poses across denoising stages and further demonstrate that early and late stages play complementary roles in forming global structure and refining local details. Extensive experiments demonstrate that DualCamCtrl achieves more consistent camera-controlled video generation, with over 40\% reduction in camera motion errors compared with prior methods. Our project page: https://soyouthinkyoucantell.github.io/dualcamctrl-page/

Figures (20)

• Figure 1: Comparison with state-of-the-art methods he2024cameractrlwan2025123 on camera-controlled video generation. Under an identical target camera trajectory and a single input image, our approach achieves the closest adherence to the camera motion and yields the best perceptual quality.
• Figure 2: Overall architecture of DualCamCtrl. DualCamCtrl adopts a dual-branch framework that simultaneously generates RGB and depth video latents from an input image and its corresponding depth map. The two latents are then element-wise added to the encoded Plücker embedding and concatenated with noise (\ref{['sec:dual_bracnh']}). Subsequently, the two modalities interact through our proposed SIGMA mechanism and fusion block (\ref{['sec:pgra']}). During training, both predictions are supervised by their respective loss functions (\ref{['sec:two_stage_training']}).
• Figure 3: (a) Illustration of modality misalignment. Independent RGB and depth latent evolution leads misalignment across frames. This motivates the design of our SIGMA strategy to establish coherent cross-modal alignment. (b) Comparison with one-way alignment. One-way alignment transfers information unidirectionally, leading to misalignment on local semantics. (c) Comparison with geometry-guided alignment Under the geometry-guided setting, geometry cues evolved too quickly and become inconsistent with RGB motion.
• Figure 4: Comparison of fusion strategies. (1.a) Previous shallow linear fusion operates in a pixel-wise manner, ignoring temporal and spatial context and causing inconsistency over time. (1.b) We introduce 3D Fusion strategy to extend the fusion formulation from 1D to 3D by incorporating 3D operations. (2) Traditional linear-layer fusion often leads to visual artifacts, while our methods produce smoother, more coherent results.
• Figure 5: CKA vs. pose; early-stage effects. (a) RGB branch shows strongest camera motion alignment in early–mid layers. (b) Extra early steps yield the largest quality gains. (c) Stronger early-stage weights lower CKA variance and improve FVD.