NIS-SLAM: Neural Implicit Semantic RGB-D SLAM for 3D Consistent Scene Understanding

TL;DR

This work addresses the challenge of achieving 3D-consistent scene understanding within neural implicit SLAM using noisy 2D segmentation. It introduces NIS-SLAM, a system that combines high-frequency tetrahedron-based features with low-frequency positional encoding, coupled with a multi-view semantic fusion strategy and confidence-guided tracking, to jointly learn geometry, appearance, and semantics from RGB-D streams. The approach delivers robust camera tracking, high-fidelity surface reconstruction, and 3D semantic fields, achieving state-of-the-art or competitive results on Replica, ScanNet, and TUM, and enabling AR applications. The work advances practical neural implicit SLAM by improving semantic consistency across views and reducing memory/compute demands through a tetrahedron-based representation and progressive optimization.

Abstract

In recent years, the paradigm of neural implicit representations has gained substantial attention in the field of Simultaneous Localization and Mapping (SLAM). However, a notable gap exists in the existing approaches when it comes to scene understanding. In this paper, we introduce NIS-SLAM, an efficient neural implicit semantic RGB-D SLAM system, that leverages a pre-trained 2D segmentation network to learn consistent semantic representations. Specifically, for high-fidelity surface reconstruction and spatial consistent scene understanding, we combine high-frequency multi-resolution tetrahedron-based features and low-frequency positional encoding as the implicit scene representations. Besides, to address the inconsistency of 2D segmentation results from multiple views, we propose a fusion strategy that integrates the semantic probabilities from previous non-keyframes into keyframes to achieve consistent semantic learning. Furthermore, we implement a confidence-based pixel sampling and progressive optimization weight function for robust camera tracking. Extensive experimental results on various datasets show the better or more competitive performance of our system when compared to other existing neural dense implicit RGB-D SLAM approaches. Finally, we also show that our approach can be used in augmented reality applications. Project page: \href{https://zju3dv.github.io/nis_slam}{https://zju3dv.github.io/nis\_slam}.

Figures (10)

• Figure 1: Our system takes RGB-D frames as input to perform camera tracking and mapping via volume rendering and models 3D semantics with the noise 2D segmentation results from Mask2Former cheng2021mask2former. Based on the hybrid implicit representation of multi-resolution tetrahedron feature $\theta$ and positional encoding $\texttt{PE}(p)$, we decode the SDF $\sigma$, latent feature $h$, color $c$, and semantic probability $s$ with three MLPs {$\mathcal{M}_{geo}$, $\mathcal{M}_{color}$, and $\mathcal{M}_{sem}$}. To model consistent semantic property, we fuse multi-view semantics of nearby non-keyframes for learning 3D consistent representation.
• Figure 2: Barycentric interpolation of multi-resolution tetrahedron feature. For tetrahedrons of different resolutions, we obtain the features of the query point according to the vertex features and barycentric.
• Figure 3: Segmentation confidence map and tracking error map. Complex geometry and ambiguous regions usually lead to lower confidence and slow convergence speed for color and geometry fields. As shown in the right figure, those regions will lead to large tracking errors.
• Figure 4: Reconstruction Results on ScanNet dai:2017:scannet. Compared to the baselines, our method can reconstruct more accurate detailed geometry and generate more complete, smoother mesh.
• Figure 5: Object reconstruction of Replica julian:2019:replica. We show some selected objects for comparison with vMAP kong2023vmap.