NexusSplats: Efficient 3D Gaussian Splatting in the Wild

TL;DR

NexusSplats introduces a scalable 3D Gaussian Splatting framework tailored for unstructured real-world scenes with complex lighting and transient occlusions. It couples hierarchical light decoupling via nexus kernels with structure-aware occlusion handling that propagates 3D uncertainties to 2D semantics and employs boundary refinement, all under a unified color–uncertainty optimization. The approach achieves state-of-the-art rendering quality while dramatically reducing parameters and speeding up training (e.g., 65.4% fewer parameters and 2.7× faster reconstruction) compared with strong baselines. The work promises practical impact for real-world 3D reconstruction in AR/VR, robotics, and autonomous navigation by delivering high-fidelity, efficiently trainable scene representations in the wild.

Abstract

Photorealistic 3D reconstruction of unstructured real-world scenes remains challenging due to complex illumination variations and transient occlusions. Existing methods based on Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) struggle with inefficient light decoupling and structure-agnostic occlusion handling. To address these limitations, we propose NexusSplats, an approach tailored for efficient and high-fidelity 3D scene reconstruction under complex lighting and occlusion conditions. In particular, NexusSplats leverages a hierarchical light decoupling strategy that performs centralized appearance learning, efficiently and effectively decoupling varying lighting conditions. Furthermore, a structure-aware occlusion handling mechanism is developed, establishing a nexus between 3D and 2D structures for fine-grained occlusion handling. Experimental results demonstrate that NexusSplats achieves state-of-the-art rendering quality and reduces the number of total parameters by 65.4\%, leading to 2.7$\times$ faster reconstruction.

Figures (13)

• Figure 1: (a) Given photos from in-the-wild scenarios, (b) our method decouples lighting conditions and eliminates occlusions, enabling steerable color mapping to diverse lighting conditions. (c) NexusSplats achieves state-of-the-art rendering quality, with a substantial training speed improvement over extensions of 3DGS.
• Figure 2: Overview of NexusSplats. Our framework operates in three stages: First, the hierarchical Gaussian management (\ref{['sec:hierarchical management']}) organizes 3D Gaussians into dynamic nexus kernels, which generate Gaussian attributes and perform centralized appearance learning (\ref{['sec:centralized appearance learning']}) and uncertainty propagation (\ref{['sec:uncertainty propagation']}). Second, a raw image $\hat{\mathbf{C}}$, a mapped image $\tilde{\mathbf{C}}$, and an uncertainty mask $\frac{1}{2\hat{\mathbf{\beta}}^2}$ are rendered through tile rasterization. Third, the boundary-aware refinement (\ref{['sec:boundary-aware refinement']}) corrects misclassified scene boundaries. The system optimizes via a combination of color loss $\mathcal{L}_c$ and uncertainty loss $\mathcal{L}_u$.
• Figure 3: Qualitative Comparison on the Photo Tourism dataset snavely2006photo. For each row, different architectural scenes are shown with highlighted red and orange boxes for closer inspection. NexusSplats presents sharper details and improved color fidelity, closely matching the ground truth images, particularly in challenging areas with intricate textures.
• Figure 4: Qualitative Comparison on the NerRF On-the-go dataset ren2024nerfonthego. We present the results of three scenes with different ratios of occlusions. Our method removes all occlusions and shows the best view synthesis results.
• Figure 5: Light Decoupling Visualizations on Photo Tourism snavely2006photo. NexusSplats maps colors from the reconstructed scene to match the target lighting conditions from ground truth images.