CamPilot: Improving Camera Control in Video Diffusion Model with Efficient Camera Reward Feedback

TL;DR

CamPilot addresses the challenge of precise camera control in video diffusion by introducing a camera-aware 3D decoder that renders per-pixel 3D Gaussians (3DGS) from video latents and Plücker camera embeddings. This decoder enables efficient, geometry-aware reward computation and feeds a visibility-aware pixel loss for reward optimization, reducing noise from stochastic generation. The proposed Camera Reward Optimization (CRO) minimizes deterministic pixel differences between rendered and ground-truth sequences, guided by depth-based visibility, and is trained end-to-end with a camera-conditioned diffusion backbone. Empirical results on RealEstate10K and WorldScore demonstrate improved camera controllability and visual fidelity, with ablations validating the contributions of ReFL, novel-view supervision, and the visibility mechanism. The work highlights a practical, scalable path to high-quality, world-consistent video generation with explicit 3D-geometry awareness.

Abstract

Recent advances in camera-controlled video diffusion models have significantly improved video-camera alignment. However, the camera controllability still remains limited. In this work, we build upon Reward Feedback Learning and aim to further improve camera controllability. However, directly borrowing existing ReFL approaches faces several challenges. First, current reward models lack the capacity to assess video-camera alignment. Second, decoding latent into RGB videos for reward computation introduces substantial computational overhead. Third, 3D geometric information is typically neglected during video decoding. To address these limitations, we introduce an efficient camera-aware 3D decoder that decodes video latent into 3D representations for reward quantization. Specifically, video latent along with the camera pose are decoded into 3D Gaussians. In this process, the camera pose not only acts as input, but also serves as a projection parameter. Misalignment between the video latent and camera pose will cause geometric distortions in the 3D structure, resulting in blurry renderings. Based on this property, we explicitly optimize pixel-level consistency between the rendered novel views and ground-truth ones as reward. To accommodate the stochastic nature, we further introduce a visibility term that selectively supervises only deterministic regions derived via geometric warping. Extensive experiments conducted on RealEstate10K and WorldScore benchmarks demonstrate the effectiveness of our proposed method. Project page: \href{https://a-bigbao.github.io/CamPilot/}{CamPilot Page}.

Figures (12)

• Figure 1: Our model functions as a comprehensive framework for world-consistent video generation and scene reconstruction. In the upper section, it excels at generating 3D-consistent scene videos for world exploration by following custom camera trajectories. In the lower section, it efficiently reconstructs high-quality 3D scenes in a feed-forward manner with generated video frames.
• Figure 2: Overall of our framework. It consists of (a): a camera-controlled I2V model, where we inject Plücker Embedding as camera condition using ControlNet. (b) A camera-aware 3D decoder that decodes latent to 3DGS, supporting rendering for reward computation. (c) Camera reward optimization that minimizes mask-aware difference between rendered videos and ground-truth ones.
• Figure 3: We add perturbation to the GT camera pose, and the rendered image becomes noticeably blurred, indicating the importance of aligned poses for rendering photorealistic images.
• Figure 4: Qualitative comparison of video generation: our model produces novel views that are better aligned with the camera poses.
• Figure 5: Qualitative comparison of 3D scene generation: our model produces more photorealistic novel view renderings that are aligned with the camera poses, outperforming other methods