Constructing a 3D Scene from a Single Image

TL;DR

This work tackles the problem of generating coherent 3D scenes from a single top-down image. It introduces SceneFuse-3D, a training-free pipeline that combines region-based latent synthesis with spatial-aware 3D inpainting guided by monocular depth and landmark priors. The method decomposes the scene into overlapping regions, completes each with a two-stage rectified-flow-based process, and fuses them to produce geometrically rich, layout-consistent 3D scenes, decoded into textured meshes and Gaussians. Extensive experiments show SceneFuse-3D surpasses state-of-the-art baselines in geometry quality, layout coherence, and texture fidelity, validated by human, GPT-4o, and rendered-view metrics. This approach offers a scalable, input-efficient pathway to realistic 3D scene synthesis suitable for simulation, robotics, and digital content creation.

Abstract

Acquiring detailed 3D scenes typically demands costly equipment, multi-view data, or labor-intensive modeling. Therefore, a lightweight alternative, generating complex 3D scenes from a single top-down image, plays an essential role in real-world applications. While recent 3D generative models have achieved remarkable results at the object level, their extension to full-scene generation often leads to inconsistent geometry, layout hallucinations, and low-quality meshes. In this work, we introduce SceneFuse-3D, a training-free framework designed to synthesize coherent 3D scenes from a single top-down view. Our method is grounded in two principles: region-based generation to improve image-to-3D alignment and resolution, and spatial-aware 3D inpainting to ensure global scene coherence and high-quality geometry generation. Specifically, we decompose the input image into overlapping regions and generate each using a pretrained 3D object generator, followed by a masked rectified flow inpainting process that fills in missing geometry while maintaining structural continuity. This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning. Extensive experiments across diverse scenes show that SceneFuse-3D outperforms state-of-the-art baselines, including Trellis, Hunyuan3D-2, TripoSG, and LGM, in terms of geometry quality, spatial coherence, and texture fidelity. Our results demonstrate that high-quality coherent 3D scene-level asset generation is achievable from a single top-down image using a principled, training-free pipeline.

Figures (6)

• Figure 1: 3D Scene Generation from a Single Image. Given a top-down reference image (center), SceneFuse-3D generates coherent and realistic 3D scenes that preserve geometry, texture, and layout compared to other state-of-the-art end-to-end image-to-3D generation models. Our method also generalizes across diverse styles (right), producing high-quality outputs without any 3D training.
• Figure 2: Overview of the SceneFuse-3D Pipeline. Given a single top-down image, we first estimate a coarse scene structure via monocular depth and landmark extraction to initialize the scene latent (Spatial Prior Initialization). The scene is divided into overlapping regions for localized synthesis and progressively fused into a coherent global latent (Region-based Generation & Fusion). Each region is completed using a two-stage masked rectified flow pipeline with a sparse structure generator $\mathcal{G}_s$ and a structured latent generator $\mathcal{G}_L$ (Spatial-aware 3D Completion). The final 3D scene is decoded from the completed structured latent.
• Figure 3: Qualitative comparisons between SceneFuse-3D and baselines. Given a single top-down image (left column), we compare 3D scene outputs generated by SceneFuse-3D, Trellis xiang2024structured, Hunyuan3D-2 hunyuan3d22025tencent, TripoSG li2025triposg, and LGM tang2024lgm. SceneFuse-3D consistently produces globally coherent scenes with fine-grained geometry, accurate object layouts, and realistic textures across a variety of styles and environments. In contrast, Trellis often produces oversimplified geometry; Hunyuan3D-2 suffers from structural inconsistencies and domain mismatch; TripoSG exhibits repetition artifacts and layout drift; LGM cannot generate consistent multi-view scene images for 3D construction.
• Figure 4: Qualitative Ablation Results. Left: Reference image. Middle: SceneFuse-3D without landmark conditioning. Right: SceneFuse-3D without region-based generation. Landmark conditioning ensures consistency for foreground objects, especially for objects across regions, while region-based generation preserves overall detail and coherence.
• Figure 5: More qualitative comparisons between SceneFuse-3D and baselines. From the image, we can find that SceneFuse-3D can generate more coherent scenes from diverse scene images. LGM is skipped since it fails to generate structured scenes for all inputs.