Scene4U: Hierarchical Layered 3D Scene Reconstruction from Single Panoramic Image for Your Immerse Exploration

TL;DR

Scene4U tackles the challenge of generating immersive, obstacle-free 3D scenes from a single panoramic image by introducing a semantic-aware, layered reconstruction pipeline. It combines a Climate Controller to create diverse target panoramas, open-vocabulary segmentation with LLM-based semantic filtering to produce hierarchical masks, occlusion-aware FLUX-inpainting and depth completion across layers, and a layer-wise 3D Gaussian Splatting (3DGS) optimization that refines the scene from back to front. The approach yields globally consistent, occlusion-free 3D scenes suitable for free navigation and immersive exploration, outperforming state-of-the-art single-view methods in both quality metrics and efficiency. A new WorldVista3D panorama dataset is provided to support robust evaluation across globally renowned landmarks, with code released for reproducibility and broader impact.

Abstract

The reconstruction of immersive and realistic 3D scenes holds significant practical importance in various fields of computer vision and computer graphics. Typically, immersive and realistic scenes should be free from obstructions by dynamic objects, maintain global texture consistency, and allow for unrestricted exploration. The current mainstream methods for image-driven scene construction involves iteratively refining the initial image using a moving virtual camera to generate the scene. However, previous methods struggle with visual discontinuities due to global texture inconsistencies under varying camera poses, and they frequently exhibit scene voids caused by foreground-background occlusions. To this end, we propose a novel layered 3D scene reconstruction framework from panoramic image, named Scene4U. Specifically, Scene4U integrates an open-vocabulary segmentation model with a large language model to decompose a real panorama into multiple layers. Then, we employs a layered repair module based on diffusion model to restore occluded regions using visual cues and depth information, generating a hierarchical representation of the scene. The multi-layer panorama is then initialized as a 3D Gaussian Splatting representation, followed by layered optimization, which ultimately produces an immersive 3D scene with semantic and structural consistency that supports free exploration. Scene4U outperforms state-of-the-art method, improving by 24.24% in LPIPS and 24.40% in BRISQUE, while also achieving the fastest training speed. Additionally, to demonstrate the robustness of Scene4U and allow users to experience immersive scenes from various landmarks, we build WorldVista3D dataset for 3D scene reconstruction, which contains panoramic images of globally renowned sites. The implementation code and dataset will be released at https://github.com/LongHZ140516/Scene4U .

Figures (7)

• Figure 1: Overview of Scene4U. Scene4U is an unobstructed 3D scene construction framework based on single-view panoramas. By inputting a real panoramic image, Scene4U reconstructs a 3D scene free from dynamic objects such as pedestrians and vehicles, supporting unrestricted navigation.
• Figure 2: The overview of Scene4U pipeline. In the first stage, we use the input panoramic image and text prompts to generate a panoramic image with corresponding spatiotemporal characteristics through Climate Controller, followed by multi-layer segmentation. In the second stage, we use the obtained multi-layer mask results to perform multi-layer construction on the panoramic scene image. In the third stage, we apply a layered training strategy to optimize the scene, reconstructing an immersive environment for free exploration.
• Figure 3: Illustration of Climate Controller synthesis results. The Climate Controller module can generate realistic street-view images under various weather and time conditions, enhancing the diversity of reconstructed scenes.
• Figure 4: The segmentation results comparison of the close-vocabulary and open-vocabulary segmentation.
• Figure 5: Illustration of multi-layer segmentation strategy. Starting with initial open-vocabulary segmentation labels for the panorama images, we utilize LLM to group categories and output masks for dynamic objects, foreground, background, and sky regions, respectively.