Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Building temporally coherent 3D maps with VGGT for memory-efficient Semantic SLAM

Gergely Dinya, Péter Halász, András Lőrincz, Kristóf Karacs, Anna Gelencsér-Horváth

TL;DR

The paper tackles the memory and latency challenges of applying VGGT-based semantic SLAM to long indoor sequences for assistive navigation. It introduces a memory-efficient, online pipeline that processes video in non-overlapping blocks to build and align submaps, fusing 2D instance segmentation with VGGT tracking to create persistent 3D objects and incorporating a lightweight change-detection mechanism via timestamps. Change states (Recent/Retained/Removed) are updated through visibility and depth-consistency checks, enabling robust detection of object presence and state changes. Evaluations on TUM RGB-D and 7-Scenes (plus custom assistive data) demonstrate competitive accuracy with significantly reduced memory footprint and near real-time performance on commodity hardware, indicating practical viability for assistive navigation scenarios.

Abstract

We present a fast, spatio-temporal scene understanding framework based on Visual Geometry Grounded Transformer (VGGT). The proposed pipeline is designed to enable efficient, close to real-time performance, supporting applications including assistive navigation. To achieve continuous updates of the 3D scene representation, we process the image flow with a sliding window, aligning submaps, thereby overcoming VGGT's high memory demands. We exploit the VGGT tracking head to aggregate 2D semantic instance masks into 3D objects. To allow for temporal consistency and richer contextual reasoning the system stores timestamps and instance-level identities, thereby enabling the detection of changes in the environment. We evaluate the approach on well-known benchmarks and custom datasets specifically designed for assistive navigation scenarios. The results demonstrate the applicability of the framework to real-world scenarios.

Building temporally coherent 3D maps with VGGT for memory-efficient Semantic SLAM

TL;DR

The paper tackles the memory and latency challenges of applying VGGT-based semantic SLAM to long indoor sequences for assistive navigation. It introduces a memory-efficient, online pipeline that processes video in non-overlapping blocks to build and align submaps, fusing 2D instance segmentation with VGGT tracking to create persistent 3D objects and incorporating a lightweight change-detection mechanism via timestamps. Change states (Recent/Retained/Removed) are updated through visibility and depth-consistency checks, enabling robust detection of object presence and state changes. Evaluations on TUM RGB-D and 7-Scenes (plus custom assistive data) demonstrate competitive accuracy with significantly reduced memory footprint and near real-time performance on commodity hardware, indicating practical viability for assistive navigation scenarios.

Abstract

We present a fast, spatio-temporal scene understanding framework based on Visual Geometry Grounded Transformer (VGGT). The proposed pipeline is designed to enable efficient, close to real-time performance, supporting applications including assistive navigation. To achieve continuous updates of the 3D scene representation, we process the image flow with a sliding window, aligning submaps, thereby overcoming VGGT's high memory demands. We exploit the VGGT tracking head to aggregate 2D semantic instance masks into 3D objects. To allow for temporal consistency and richer contextual reasoning the system stores timestamps and instance-level identities, thereby enabling the detection of changes in the environment. We evaluate the approach on well-known benchmarks and custom datasets specifically designed for assistive navigation scenarios. The results demonstrate the applicability of the framework to real-world scenarios.

Paper Structure

This paper contains 14 sections, 17 equations, 4 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Methods
  4. Global alignment
  5. 3D segmentation
  6. Change detection
  7. Ego-location and object positions
  8. Results
  9. Experimental setup and metrics
  10. Quantitative results
  11. Qualitative results
  12. Discussion
  13. Conclusion
  14. Acknowledgments

Figures (4)

  • Figure 1: Overview of the proposed pipeline: memory-efficient, temporally coherent semantic SLAM with VGGT.
  • Figure 2: The input stream is partitioned into blocks, with key frames serving as anchors for alignment.
  • Figure 3: Tracked points sampled from instance masks. Each color corresponds to the assigned instance mask, while dark blue dots indicate unassigned (propagated) points.
  • Figure 4: Point-cloud comparison on identical inputs across four sequences. Top row: VGGT– SLAM; bottom row: our method. Vertically aligned pairs correspond to the same input.