Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FeatureSLAM: Feature-enriched 3D gaussian splatting SLAM in real time

Christopher Thirgood, Oscar Mendez, Erin Ling, Jon Storey, Simon Hadfield

TL;DR

FeatureSLAM addresses real-time SLAM with open-set semantic understanding by embedding multi-scale foundation-model features into a 3D Gaussian Splatting (3DGS) map and jointly optimizing rendering, geometry, and semantics online. The approach introduces online feature rasterization, camera-plane depth rasterization, depth-normal regularization, and per-splat parallelization, coupled with a semantic GICP objective and gradient-based pruning to maintain a compact yet expressive map. It enables promptless segmentation and language-guided queries via Grounding-SAM2, achieving competitive tracking and improved novel-view semantics without offline preprocessing. Experimental results on Replica and TUM-RGBD demonstrate real-time performance (≈5 FPS) with strong pose accuracy and high-quality reconstructions, confirming the practicality and downstream potential of feature-enriched SLAM for robotics and AR applications.

Abstract

We present a real-time tracking SLAM system that unifies efficient camera tracking with photorealistic feature-enriched mapping using 3D Gaussian Splatting (3DGS). Our main contribution is integrating dense feature rasterization into the novel-view synthesis, aligned with a visual foundation model. This yields strong semantics, going beyond basic RGB-D input, aiding both tracking and mapping accuracy. Unlike previous semantic SLAM approaches (which embed pre-defined class labels) FeatureSLAM enables entirely new downstream tasks via free-viewpoint, open-set segmentation. Across standard benchmarks, our method achieves real-time tracking, on par with state-of-the-art systems while improving tracking stability and map fidelity without prohibitive compute. Quantitatively, we obtain 9\% lower pose error and 8\% higher mapping accuracy compared to recent fixed-set SLAM baselines. Our results confirm that real-time feature-embedded SLAM, is not only valuable for enabling new downstream applications. It also improves the performance of the underlying tracking and mapping subsystems, providing semantic and language masking results that are on-par with offline 3DGS models, alongside state-of-the-art tracking, depth and RGB rendering.

FeatureSLAM: Feature-enriched 3D gaussian splatting SLAM in real time

TL;DR

FeatureSLAM addresses real-time SLAM with open-set semantic understanding by embedding multi-scale foundation-model features into a 3D Gaussian Splatting (3DGS) map and jointly optimizing rendering, geometry, and semantics online. The approach introduces online feature rasterization, camera-plane depth rasterization, depth-normal regularization, and per-splat parallelization, coupled with a semantic GICP objective and gradient-based pruning to maintain a compact yet expressive map. It enables promptless segmentation and language-guided queries via Grounding-SAM2, achieving competitive tracking and improved novel-view semantics without offline preprocessing. Experimental results on Replica and TUM-RGBD demonstrate real-time performance (≈5 FPS) with strong pose accuracy and high-quality reconstructions, confirming the practicality and downstream potential of feature-enriched SLAM for robotics and AR applications.

Abstract

We present a real-time tracking SLAM system that unifies efficient camera tracking with photorealistic feature-enriched mapping using 3D Gaussian Splatting (3DGS). Our main contribution is integrating dense feature rasterization into the novel-view synthesis, aligned with a visual foundation model. This yields strong semantics, going beyond basic RGB-D input, aiding both tracking and mapping accuracy. Unlike previous semantic SLAM approaches (which embed pre-defined class labels) FeatureSLAM enables entirely new downstream tasks via free-viewpoint, open-set segmentation. Across standard benchmarks, our method achieves real-time tracking, on par with state-of-the-art systems while improving tracking stability and map fidelity without prohibitive compute. Quantitatively, we obtain 9\% lower pose error and 8\% higher mapping accuracy compared to recent fixed-set SLAM baselines. Our results confirm that real-time feature-embedded SLAM, is not only valuable for enabling new downstream applications. It also improves the performance of the underlying tracking and mapping subsystems, providing semantic and language masking results that are on-par with offline 3DGS models, alongside state-of-the-art tracking, depth and RGB rendering.
Paper Structure (33 sections, 30 equations, 10 figures, 10 tables)

This paper contains 33 sections, 30 equations, 10 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Localization in SLAM
  4. Mapping in SLAM
  5. 3D Gaussian Splatting.
  6. Feature distilled 3DGS
  7. Preliminaries
  8. 3DGS Rendering
  9. GICP Tracking
  10. Correspondence and Gaussian error model
  11. Maximum-likelihood pose estimation.
  12. Methodology
  13. Online Feature Rasterization
  14. Robust Depth Rasterization.
  15. Depth-normal consistency & regularization
  16. ...and 18 more sections

Figures (10)

  • Figure 1: FeatureSLAM overview, including the feature extraction, tracking and mapping modules.
  • Figure 2: Qualitative comparison of novel-views Replica-office0.
  • Figure 3: Screenshots from the live demonstration and free camera viewer
  • Figure 4: Promptless segmentation result with overlay visualisation from GT rgb image.
  • Figure 5: Promptless segmentation result with overlay visualisation from GT rgb image.
  • ...and 5 more figures