Free360: Layered Gaussian Splatting for Unbounded 360-Degree View Synthesis from Extremely Sparse and Unposed Views

TL;DR

This work tackles unposed, extremely sparse-view 360° scene reconstruction. It introduces Free360, a framework that ensembles layered Gaussian Splatting with layer-specific bootstrap optimization and an iterative reconstruction-generation loop powered by a diffusion model, all guided by uncertainty-aware training. Across Mip-NeRF 360 and Tanks and Temples, Free360 achieves superior rendering quality and geometry accuracy with as few as 3–4 input views, demonstrating robust performance against pose noise and sparse observations. The approach provides a practical, modular pathway to high-quality unbounded scene reconstruction from minimal, unposed data, with potential for integration with stronger generative priors in the future.

Abstract

Neural rendering has demonstrated remarkable success in high-quality 3D neural reconstruction and novel view synthesis with dense input views and accurate poses. However, applying it to extremely sparse, unposed views in unbounded 360° scenes remains a challenging problem. In this paper, we propose a novel neural rendering framework to accomplish the unposed and extremely sparse-view 3D reconstruction in unbounded 360° scenes. To resolve the spatial ambiguity inherent in unbounded scenes with sparse input views, we propose a layered Gaussian-based representation to effectively model the scene with distinct spatial layers. By employing a dense stereo reconstruction model to recover coarse geometry, we introduce a layer-specific bootstrap optimization to refine the noise and fill occluded regions in the reconstruction. Furthermore, we propose an iterative fusion of reconstruction and generation alongside an uncertainty-aware training approach to facilitate mutual conditioning and enhancement between these two processes. Comprehensive experiments show that our approach outperforms existing state-of-the-art methods in terms of rendering quality and surface reconstruction accuracy. Project page: https://zju3dv.github.io/free360/

Figures (12)

• Figure 1: Free360. (a) We propose a novel Gaussian-based framework, which can reconstruct unbounded 360$^{\circ}$ scenes from extremely (3-4) sparse views through an iterative fusion of layered reconstruction and generation. (b) Our method outperforms other state-of-the-art methods in rendering quality and supports complete surface reconstruction.
• Figure 2: Pipeline. (a-d) Given unposed extremely sparse views, we employ the dense stereo reconstruction model dust3rmast3r to recover camera poses and initial point cloud of the scene. A layered Gaussian-based representation is built upon the initial point cloud to enable layer-specific bootstrap optimization. (e) We design the iterative fusion of reconstruction and generation with diffusion model yu2024viewcrafter. Unknown views are iteratively generated under conditions of consistent GS rendering of known views. In turn, generated views are used to enhance the GS training.
• Figure 3: Visualization of Point Clouds. We compare results from the stereo reconstruction model and our bootstrap optimization.
• Figure 4: Uncertainty Map. We show the uncertainty map estimated from generated novel views. The flower has severe multi-view inconsistency with high uncertainty.
• Figure 5: Comparison on Mip-NeRF 360$^{\circ}$barron2022mipnerf360 Dataset on the 3-View Setting. We qualitatively compare rendering quality with FSGS* fsgs, InstantSplat fan2024instantsplat, ZeroNVS* sargent2024zeronvs, ViewCrafter yu2024viewcrafter given 3 input views.