VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control

TL;DR

VerseCrafter tackles the challenge of controllable, realistic video generation by introducing a 4D Geometric Control that unifies camera and multi-object dynamics in a shared 4D space. It maps this 4D state to video via a lightweight GeoAdapter that conditions a frozen Wan2.1 diffusion backbone, enabling view-consistent generation aligned with explicit 4D instructions. The authors also release VerseControl4D, a large-scale real-world dataset with automatically annotated 4D controls to train and evaluate the model on diverse scenes. The approach yields higher visual quality and more accurate 3D motion control than prior methods, highlighting the potential of explicit 4D state representations for dynamic world editing and synthesis.

Abstract

Video world models aim to simulate dynamic, real-world environments, yet existing methods struggle to provide unified and precise control over camera and multi-object motion, as videos inherently operate dynamics in the projected 2D image plane. To bridge this gap, we introduce VerseCrafter, a 4D-aware video world model that enables explicit and coherent control over both camera and object dynamics within a unified 4D geometric world state. Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state through a static background point cloud and per-object 3D Gaussian trajectories. This representation captures not only an object's path but also its probabilistic 3D occupancy over time, offering a flexible, category-agnostic alternative to rigid bounding boxes or parametric models. These 4D controls are rendered into conditioning signals for a pretrained video diffusion model, enabling the generation of high-fidelity, view-consistent videos that precisely adhere to the specified dynamics. Unfortunately, another major challenge lies in the scarcity of large-scale training data with explicit 4D annotations. We address this by developing an automatic data engine that extracts the required 4D controls from in-the-wild videos, allowing us to train our model on a massive and diverse dataset.

Figures (14)

• Figure 1: VerseCrafter enables precise control of camera motion and multi-object motion via a 4D Geometric Control representation built from a static background point cloud and per-object 3D Gaussian trajectories, producing videos that better follow the desired motion than Yume mao2025yume and Uni3C cao2025uni3c and closely match the ground-truth video.
• Figure 2: Framework of VerseCrafter. Given an input image and a text prompt, we first estimate depth and obtain user-specified object masks to construct a 4D Geometric Control state consisting of a static background point cloud and per-object 3D Gaussian trajectories in a shared world frame. This state is rendered into background RGB/depth, 3D Gaussian trajectory RGB/depth, and a soft control mask for each frame, forming multi-channel 4D control maps. The control maps are encoded and fed into the proposed GeoAdapter, which conditions a frozen Wan2.1-14B video diffusion backbone together with text embeddings from umT5, enabling geometry-consistent video generation with precise control over camera and multi-object motion.
• Figure 3: Starting from Sekai-Real-HQ and SpatialVID-HQ, we obtain 81-frame clips extraction, followed by quality filtering. For each retained clip, Qwen2.5-VL-72B, Grounded-SAM2, and MegaSAM provide captions, object masks, depth, and camera poses, which are lifted into background/object point clouds, fitted with 3D Gaussian trajectories, and rendered as background/trajectory maps plus a merged mask that constitute our 4D Geometric Control.
• Figure 4: Qualitative comparison of joint camera and object motion control. Perception-as-Control often yields low-fidelity frames with inaccurate camera motion, Yume roughly follows the text-described motion but lacks precise control, and Uni3C is limited to human motion. VerseCrafter more faithfully follows both the camera trajectory and multi-object motion while maintaining sharp appearance and geometrically consistent backgrounds.
• Figure 5: Qualitative comparison of camera-only motion control on static scenes. ViewCrafter, Voyager, and FlashWorld often exhibit distorted facades, drifting structures, or inconsistent parallax along the camera path. VerseCrafter better follows the target trajectory while preserving sharp details and globally consistent 3D geometry.