GeoDiff3D: Self-Supervised 3D Scene Generation with Geometry-Constrained 2D Diffusion Guidance

TL;DR

GeoDiff3D tackles the problem of efficient, high-fidelity 3D scene generation under limited labeled data by coupling coarse geometric structure with a geometry-constrained 2D diffusion prior for texture synthesis. It introduces a three-stage pipeline: (i) texture reference image generation from coarse geometry and diffusion with line-prior guidance, (ii) voxel-aligned 3D feature aggregation that back-projects multi-view textures into a sparse voxel grid and decodes to voxel-aligned Gaussians (with $J=32$ per voxel), and (iii) a dual self-supervised optimization that enforces cross-view consistency while preserving high-frequency details. Key contributions include the voxel-aligned aggregation mechanism, a bounded Gaussian parameterization with a deterministic perturbation to reduce artifacts, and a dual supervision strategy that reduces the need for large 3D supervision. The approach demonstrates improved generalization and generation quality over baselines on challenging scenes and runs efficiently on modest hardware, enabling accessible and rapid 3D content creation. Overall, GeoDiff3D provides a practical framework for geometry-guided diffusion-assisted 3D scene generation with strong structural coherence and texture fidelity.

Abstract

3D scene generation is a core technology for gaming, film/VFX, and VR/AR. Growing demand for rapid iteration, high-fidelity detail, and accessible content creation has further increased interest in this area. Existing methods broadly follow two paradigms - indirect 2D-to-3D reconstruction and direct 3D generation - but both are limited by weak structural modeling and heavy reliance on large-scale ground-truth supervision, often producing structural artifacts, geometric inconsistencies, and degraded high-frequency details in complex scenes. We propose GeoDiff3D, an efficient self-supervised framework that uses coarse geometry as a structural anchor and a geometry-constrained 2D diffusion model to provide texture-rich reference images. Importantly, GeoDiff3D does not require strict multi-view consistency of the diffusion-generated references and remains robust to the resulting noisy, inconsistent guidance. We further introduce voxel-aligned 3D feature aggregation and dual self-supervision to maintain scene coherence and fine details while substantially reducing dependence on labeled data. GeoDiff3D also trains with low computational cost and enables fast, high-quality 3D scene generation. Extensive experiments on challenging scenes show improved generalization and generation quality over existing baselines, offering a practical solution for accessible and efficient 3D scene construction.

Figures (7)

• Figure 1: Overview of GeoDiff3D. In the first stage, we extract structural edges from the input 3D model along the camera trajectory and generate multi-view pseudo-GT images guided by an image diffusion prior. In the second stage, we back-project 2D features from the selected pseudo-GT views into 3D space to form a voxel feature volume, which is then decoded into a 3D Gaussian (3DGS) representation. In the third stage, we perform self-supervised optimization of the generated 3D scene using reconstruction loss, depth loss, and GAN loss.
• Figure 2: Qualitative results on the effectiveness of our self-supervised optimization strategy. Orange boxes indicate the same regions of the input geometry.
• Figure 3: Qualitative results with and without the learnable features. Orange boxes indicate the same regions of the input geometry.
• Figure 4: Qualitative results across various scenes. The first column presents the inputs, including the style reference (top-left) and the text prompt (bottom-left). The second column shows representative pseudo-GT images generated by our method. The remaining images show our renderings from different viewpoints.
• Figure 5: Qualitative comparisons with Marble MarbleWorldLabs (indoor pipeline). Orange boxes highlight Marble's geometric deviations.