GenXD: Generating Any 3D and 4D Scenes

TL;DR

GenXD introduces a unified diffusion-based framework for 3D and 4D scene generation, supported by CamVid-30K, a large-scale real-world 4D dataset curated from videos with camera pose and object motion annotations. The model employs multiview-temporal modules and mask latent conditioning to learn from both 3D and 4D data, and lifts generated views into explicit 3D representations via 3D-Gaussian Splatting and Zip-NeRF (and 4D-Gaussian Splatting for dynamics). Empirical results show competitive or superior performance across 3D and 4D tasks, including few-view reconstruction and camera-conditioned video generation, with ablations validating the importance of motion disentanglement and joint training. The work provides a practical path toward real-world, scene-level 3D/4D generation and highlights data-driven challenges and ethical considerations in realistic video synthesis.

Abstract

Recent developments in 2D visual generation have been remarkably successful. However, 3D and 4D generation remain challenging in real-world applications due to the lack of large-scale 4D data and effective model design. In this paper, we propose to jointly investigate general 3D and 4D generation by leveraging camera and object movements commonly observed in daily life. Due to the lack of real-world 4D data in the community, we first propose a data curation pipeline to obtain camera poses and object motion strength from videos. Based on this pipeline, we introduce a large-scale real-world 4D scene dataset: CamVid-30K. By leveraging all the 3D and 4D data, we develop our framework, GenXD, which allows us to produce any 3D or 4D scene. We propose multiview-temporal modules, which disentangle camera and object movements, to seamlessly learn from both 3D and 4D data. Additionally, GenXD employs masked latent conditions to support a variety of conditioning views. GenXD can generate videos that follow the camera trajectory as well as consistent 3D views that can be lifted into 3D representations. We perform extensive evaluations across various real-world and synthetic datasets, demonstrating GenXD's effectiveness and versatility compared to previous methods in 3D and 4D generation.

Figures (10)

• Figure 1: Gen$\mathcal{X}$D is a unified model for high-quality 3D and 4D generation from any number of condition images. By controlling the motion strength and condition masks, Gen$\mathcal{X}$D can support various application without any modification. The condition images are shown with star icon and the time dimension is illustrated with dash line.
• Figure 2: The pipeline for CamVid-30K data curation, including (a) camera pose estimation and (b) object motion estimation. We first leverage mask-based SfM (masks are overlayed to images in (a) for visualization) to estimate camera pose and reconstruct 3D point clouds of static parts. Then relative depth is aligned with the sparse depth and project the tracking keypoints to consecutive frame for object motion estimation.
• Figure 3: Examples for object motion estimation. The motion strength is multiplied by 100. In the first example, the girl is dancing, together with the camera moving. In the second example, the camera is zooming in (red rectangle for better illustration) but the object is static. In this case, the motion strength is much smaller.
• Figure 4: The framework of Gen$\mathcal{X}$D. We leverage mask latent conditioned diffusion model to generate 3D and 4D samples with both camera (colorful map) and image (binary map) conditions. In addition, multiview-temporal modules together with $\alpha$-fusing are proposed to effectively disentangle and fuse multiview and temporal information.
• Figure 5: Qualitative comparison with camera conditioned video generation methods. Gen$\mathcal{X}$D can generate video well-aligned with camera trajectory and containing realistic object motion. (Please refer to supplementary video for better illustration.)