Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation

TL;DR

Voyager presents a world-consistent RGB-D video diffusion framework that enables explorable 3D scene generation from a single image along user-defined camera trajectories. It introduces geometry-informed frame conditioning, depth-fused diffusion, and a scalable world cache with point culling to sustain long-range, coherent exploration, complemented by a scalable data engine for automatic camera and metric-depth annotation. The approach yields improved visual fidelity and geometric accuracy, supporting direct 3D reconstruction and infinite world expansion. Empirical results across video, scene, and world-generation benchmarks demonstrate state-of-the-art performance and strong applicability to 3D content creation and virtual-world applications.

Abstract

Real-world applications like video gaming and virtual reality often demand the ability to model 3D scenes that users can explore along custom camera trajectories. While significant progress has been made in generating 3D objects from text or images, creating long-range, 3D-consistent, explorable 3D scenes remains a complex and challenging problem. In this work, we present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image with user-defined camera path. Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames, eliminating the need for 3D reconstruction pipelines (e.g., structure-from-motion or multi-view stereo). Our method integrates three key components: 1) World-Consistent Video Diffusion: A unified architecture that jointly generates aligned RGB and depth video sequences, conditioned on existing world observation to ensure global coherence 2) Long-Range World Exploration: An efficient world cache with point culling and an auto-regressive inference with smooth video sampling for iterative scene extension with context-aware consistency, and 3) Scalable Data Engine: A video reconstruction pipeline that automates camera pose estimation and metric depth prediction for arbitrary videos, enabling large-scale, diverse training data curation without manual 3D annotations. Collectively, these designs result in a clear improvement over existing methods in visual quality and geometric accuracy, with versatile applications.

Figures (13)

• Figure 1: Partial RGB images and partial depth maps rendered from point clouds at different frames. In scenarios involving complex occlusion relationships, partial RGB images often exhibit significant visual artifacts. In contrast, partial depth maps can accurately represent occlusions.
• Figure 2: Overview of Voyager: Given the input image and camera trajectories, we first render partial RGB images and depth maps for each viewpoint as the condition for video generation. Our world-consistent video diffusion model is trained to generate RGB-D frames simultaneously, thus supporting the direct reconstruction of the 3D world. The projected points are store in our world cache efficiently, which can be rendered as condition for the next round generation.
• Figure 3: Qualitative results on video generation. Compared to the baselines, our model can generate a more reasonable unseen region and meanwhile preserve the content in the input view.
• Figure 4: Qualitative results on Gaussian Splatting reconstruction. Our results present much more details than the compared baselines.
• Figure 5: Applications: (a) Long-range video generation. (b) Image-to-3D generation. (c) World-consistent video style transfer. (d) Monocular video depth estimation.