Gen3R: 3D Scene Generation Meets Feed-Forward Reconstruction

TL;DR

Gen3R addresses the challenge of high-fidelity 3D scene generation by uniting geometric priors from a feed-forward reconstruction model (VGGT) with RGB priors from a pre-trained video diffusion model. It introduces a geometry adapter that maps VGGT tokens into the diffusion latent space and imposes distribution alignment with appearance latents, enabling a unified latent space for joint generation. By concatenating appearance and geometry latents and fine-tuning a diffusion backbone, Gen3R can generate temporally coherent RGB videos alongside globally consistent 3D geometry, including point clouds, depth maps, and camera parameters, under various conditioning signals. Extensive experiments across real-world and synthetic datasets demonstrate state-of-the-art results in 1- and 2-view settings and show that the joint appearance-geometry modeling improves both generation quality and reconstruction robustness, highlighting mutual benefits of coupling reconstruction priors with generative priors.

Abstract

We present Gen3R, a method that bridges the strong priors of foundational reconstruction models and video diffusion models for scene-level 3D generation. We repurpose the VGGT reconstruction model to produce geometric latents by training an adapter on its tokens, which are regularized to align with the appearance latents of pre-trained video diffusion models. By jointly generating these disentangled yet aligned latents, Gen3R produces both RGB videos and corresponding 3D geometry, including camera poses, depth maps, and global point clouds. Experiments demonstrate that our approach achieves state-of-the-art results in single- and multi-image conditioned 3D scene generation. Additionally, our method can enhance the robustness of reconstruction by leveraging generative priors, demonstrating the mutual benefit of tightly coupling reconstruction and generative models.

Figures (14)

• Figure 1: Gen3R bridges foundational reconstruction models with 2D video diffusion, enabling the joint generation of 2D videos and their corresponding geometry in various settings.
• Figure 2: Method.Left: We recast an advanced transformer-based feed-forward reconstruction model, VGGT, as a VAE to produce geometry latents $\mathcal{G}$ by training an adapter on its latent tokens. The training is supervised with a reconstruction loss $\mathcal{L}_{\mathrm{rec}}$, along with a regularization term $\mathcal{L}_{\mathrm{KL}}$ that aligns $\mathcal{G}$ with the appearance latent $\mathcal{A}$, which is obtained from the VAE of a pre-trained video diffusion model, WAN. Right: We fine-tune the video diffusion model to jointly generate geometry and appearance latents, $\mathcal{Z}=[\mathcal{A}; \mathcal{G}]$, under various conditioning signals. At inference, varying the conditioning enables the generation of RGB videos and multiple 3D quantities, including global point clouds, depth maps, and camera parameters, from a single or multiple frames, as well as performing reconstruction.
• Figure 3: Qualitative Comparison of Geometry Generation in the 1-view based setting.
• Figure 4: Qualitative Comparison of Novel View Synthesis with 2-view conditions. The input images are shown on the left, and error maps are displayed overlaid on the results. Bluer colors indicate smaller errors, while redder colors indicate larger errors.
• Figure 5: Quali. Comparison of Geometry Reconstruction.