GSFF-SLAM: 3D Semantic Gaussian Splatting SLAM via Feature Field

TL;DR

This work tackles the challenge of robust online semantic SLAM under noisy or sparse 2D priors by introducing GSFF-SLAM, a dense reconstruction framework based on 3D Gaussian Splatting with per-point semantic feature embeddings. By decoupling semantic optimization from geometry and employing N-dimensional feature fields, the system can fuse various supervision signals, including ground-truth labels and text-guided priors from foundation models, to render dense semantic maps alongside accurate geometry. The approach achieves state-of-the-art semantic segmentation on Replica with 95.03% mIoU under GT priors and up to 2.9x speedups with minimal performance loss, while maintaining competitive SLAM metrics across Replica, ScanNet, and TUM-RGBD. Overall, GSFF-SLAM demonstrates robust online tracking, high-quality renderings, and flexible semantic supervision, enabling efficient online semantic reconstruction in challenging real-world scenes.

Abstract

Semantic-aware 3D scene reconstruction is essential for autonomous robots to perform complex interactions. Semantic SLAM, an online approach, integrates pose tracking, geometric reconstruction, and semantic mapping into a unified framework, shows significant potential. However, existing systems, which rely on 2D ground truth priors for supervision, are often limited by the sparsity and noise of these signals in real-world environments. To address this challenge, we propose GSFF-SLAM, a novel dense semantic SLAM system based on 3D Gaussian Splatting that leverages feature fields to achieve joint rendering of appearance, geometry, and N-dimensional semantic features. By independently optimizing feature gradients, our method supports semantic reconstruction using various forms of 2D priors, particularly sparse and noisy signals. Experimental results demonstrate that our approach outperforms previous methods in both tracking accuracy and photorealistic rendering quality. When utilizing 2D ground truth priors, GSFF-SLAM achieves state-of-the-art semantic segmentation performance with 95.03\% mIoU, while achieving up to 2.9$\times$ speedup with only marginal performance degradation.

Figures (5)

• Figure 1: Our GSFF-SLAM leverages different forms of signals to enhance various downstream online tasks. Our method projects 2D priors into the 3D feature field, enabling high-precision close-set segmentation, text-guided segmentation, and dense feature map rendering.
• Figure 2: Overview of GSFF-SLAM. Our method takes an RGB-D stream as input, leveraging 3D Gaussian Splatting with semantic feature embedding $f$ to generate RGB images, depth images, and dense feature maps. Semantic signals, derived from foundation models or ground truth, supervise the learning process, while the feature embedding $f$ is optimized independently.
• Figure 3: Qualitative comparison on rendering quality of baseline and our method. We select 4 scenes of Replica dataset and highlighted the differences with red color boxes.
• Figure 4: Qualitative comparison of semantic reconstruction performance using ground truth labels on the Replica dataset straub2019replica.
• Figure 5: Qualitative comparison of semantic reconstruction performance using noisy textual labels on the Replica dataset straub2019replica. We merge semantically similar objects with high 3D spatial overlap, such as windows and blinds, rugs and floors.