RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control

TL;DR

The paper tackles scale inconsistency and usability barriers in camera-trajectory-guided image-to-video generation by integrating monocular metric-depth estimation to build a metric-scale 3D scene used for training and inference. It introduces metric-scale alignment between depth-based reconstructions and SfM data, an interactive 3D interface for precise trajectory design with real-time previews, and a scene-constrained noise shaping mechanism to guide early diffusion stages. The approach yields significant improvements in video quality and controllability on RealEstate10K and generalizes to out-of-domain images, while enabling looping and frame interpolation applications. This work advances practical, precise, real-world camera control in diffusion-based I2V systems.

Abstract

Recent advancements in camera-trajectory-guided image-to-video generation offer higher precision and better support for complex camera control compared to text-based approaches. However, they also introduce significant usability challenges, as users often struggle to provide precise camera parameters when working with arbitrary real-world images without knowledge of their depth nor scene scale. To address these real-world application issues, we propose RealCam-I2V, a novel diffusion-based video generation framework that integrates monocular metric depth estimation to establish 3D scene reconstruction in a preprocessing step. During training, the reconstructed 3D scene enables scaling camera parameters from relative to metric scales, ensuring compatibility and scale consistency across diverse real-world images. In inference, RealCam-I2V offers an intuitive interface where users can precisely draw camera trajectories by dragging within the 3D scene. To further enhance precise camera control and scene consistency, we propose scene-constrained noise shaping, which shapes high-level noise and also allows the framework to maintain dynamic and coherent video generation in lower noise stages. RealCam-I2V achieves significant improvements in controllability and video quality on the RealEstate10K and out-of-domain images. We further enables applications like camera-controlled looping video generation and generative frame interpolation. Project page: https://zgctroy.github.io/RealCam-I2V.

Figures (11)

• Figure 1: RealCam-I2V pipeline. For training, we align camera parameters from relative scale to metric scale. For inference, we use metric depth estimation to construct the point cloud for users to interactively draw the camera trajectory. Due to the metric scale alignment, the user-given camera trajectory in the 3D scene shares the same scene scale as those in real world.
• Figure 2: Scene scale mismatch. Point clouds reconstructed from metric depth estimation (RGB) are robust and unified, whereas SfM reconstructions (yellow) are relative-scale that may vary across frames. Our alignment enables relative-to-metric conversion of scene scale for real-world applications.
• Figure 3: Camera trajectory ambiguity. The relative scene-scale measurement fundamentally hinders models from learning physically consistent camera motion.
• Figure 4: One-round generation versus multi-round generation. Our framework decouples camera adjustment via interactive 3D scenes, enabling fast feedback before slow generation.
• Figure 5: Pixel selection for noise shaping. Selected pixels are pasted onto the clean part (predicted $z_0$) of noised latent $z_t$. Empirically, applying noise shaping at high noise levels $t>0.9$ achieves the trade-off of camera control and dynamics.