SpatialSplat: Efficient Semantic 3D from Sparse Unposed Images

TL;DR

SpatialSplat addresses semantic 3D reconstruction from sparse, unposed images by introducing a dual-field semantic representation and a selective Gaussian mechanism that prune redundant Gaussians, achieving a 60% reduction in scene representation parameters without any 3D supervision. The framework leverages a ViT-based geometry predictor and lightweight 2D feature guidance to jointly learn instance priors and uncompressed semantics, enabling open-vocabulary 3D segmentation and robust novel-view synthesis. It demonstrates state-of-the-art performance on semantic 3D tasks while significantly reducing storage and computation compared to prior feed-forward methods. This approach enhances generalization and practicality for semantic 3D reconstruction from sparse imagery in real-world scenarios.

Abstract

A major breakthrough in 3D reconstruction is the feedforward paradigm to generate pixel-wise 3D points or Gaussian primitives from sparse, unposed images. To further incorporate semantics while avoiding the significant memory and storage costs of high-dimensional semantic features, existing methods extend this paradigm by associating each primitive with a compressed semantic feature vector. However, these methods have two major limitations: (a) the naively compressed feature compromises expressiveness, affecting the model's ability to capture fine-grained semantics, and (b) the pixel-wise primitive prediction introduces redundancy in overlapping areas, causing unnecessary memory overhead. To this end, we introduce \textbf{SpatialSplat}, a feedforward framework that produces redundancy-aware Gaussians and capitalizes on a dual-field semantic representation. Particularly, with the insight that primitives within the same instance exhibit high semantic consistency, we decompose the semantic representation into a coarse feature field that encodes uncompressed semantics with minimal primitives, and a fine-grained yet low-dimensional feature field that captures detailed inter-instance relationships. Moreover, we propose a selective Gaussian mechanism, which retains only essential Gaussians in the scene, effectively eliminating redundant primitives. Our proposed Spatialsplat learns accurate semantic information and detailed instances prior with more compact 3D Gaussians, making semantic 3D reconstruction more applicable. We conduct extensive experiments to evaluate our method, demonstrating a remarkable 60\% reduction in scene representation parameters while achieving superior performance over state-of-the-art methods. The code is available at https://github.com/shengyuuu/SpatialSplat.git

Figures (7)

• Figure 1: Comparison between previous methods and our SpatialSplat. (a): Previous methods predict pixel-wise Gaussians, associating each primitive with compressed semantic feature. (b): Our SpatialSplat avoids generating redundant Gaussian primitives for overlapping pixels, and it represents semantics with a dual-field architecture to better preserve the information. (c): SpatialSplat outperforms the state-of-the-art method LSM lsm for both novel-view rendering quality and semantic segmentation while being more efficient.
• Figure 2: Pipeline of SpatialSplat. The SpatialSplat processes unposed images along with their intrinsics through a 3D geometry transformer. The extracted features from the geometry transformer and the pretrained 2D model are then fed into separate Dense Prediction Transformer (DPT) heads to predict different Gaussian attributes, resulting in a fine-grained instance-aware radiance field $\boldsymbol{\mathcal{F}_I}$ and a coarse semantic feature field $\boldsymbol{\mathcal{F}_S}$. This enables the synthesis of RGB and feature maps from novel viewpoints.
• Figure 3: Qualitative comparison in NVS. SpatialSplat can synthesize realistic novel views. In challenging cases where LSM fails, such as the table legs in the first two rows and the corners in the last two rows, our method achieves significantly better results.
• Figure 4: Qualitative comparison in OVS. SpatialSplat achieves sharper and more precise segmentation results compared to previous methods. Notably, our method excels in challenging details, such as distinguishing table legs from cabinet legs (the second the fourth rows), benefiting from our dual-field architecture that captures detailed instance information and uncompressed semantics.
• Figure 5: The rendered instance features. SpatialSplat predicts clear and consistent instance features across different views.