SceneVGGT: VGGT-based online 3D semantic SLAM for indoor scene understanding and navigation

TL;DR

This work presents SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation and achieves competitive point-cloud performance on the ScanNet++ benchmark.

Abstract

We present SceneVGGT, a spatio-temporal 3D scene understanding framework that combines SLAM with semantic mapping for autonomous and assistive navigation. Built on VGGT, our method scales to long video streams via a sliding-window pipeline. We align local submaps using camera-pose transformations, enabling memory- and speed-efficient mapping while preserving geometric consistency. Semantics are lifted from 2D instance masks to 3D objects using the VGGT tracking head, maintaining temporally coherent identities for change detection. As a proof of concept, object locations are projected onto an estimated floor plane for assistive navigation. The pipeline's GPU memory usage remains under 17 GB, irrespectively of the length of the input sequence and achieves competitive point-cloud performance on the ScanNet++ benchmark. Overall, SceneVGGT ensures robust semantic identification and is fast enough to support interactive assistive navigation with audio feedback.

Figures (2)

• Figure 1: RGB and semantic reconstruction of an in-the-wild scene in an office along with the navigation map and route.
• Figure 2: Illustration of the mask sampling and point tracking among frames.