VGGT-SLAM 2.0: Real time Dense Feed-forward Scene Reconstruction

TL;DR

VGGT-SLAM 2.0 tackles drift and planar degeneracy in VGGT-SLAM by introducing a submap-based, SL(4)-manifold optimization framework that enforces identical overlapping frames and calibrations with a shared scale. It further leverages VGGT's attention layer to verify potential loop-closures in a data-driven, training-free manner, enabling more robust closed-loop operations. The approach supports open-set semantic tasks and real-time onboard execution, demonstrated across diverse environments and achieving state-of-the-art pose accuracy on TUM RGB-D with a ~ $23\%$ reduction in pose error relative to VGGT-SLAM. The work offers a practical, plug-in enhancement to feed-forward SLAM, balancing geometric rigour with scalable, dense reconstructions suitable for robotics.

Abstract

We present VGGT-SLAM 2.0, a real time RGB feed-forward SLAM system which substantially improves upon VGGT-SLAM for incrementally aligning submaps created from VGGT. Firstly, we remove high-dimensional 15-degree-of-freedom drift and planar degeneracy from VGGT-SLAM by creating a new factor graph design while still addressing the reconstruction ambiguity of VGGT given unknown camera intrinsics. Secondly, by studying the attention layers of VGGT, we show that one of the layers is well suited to assist in image retrieval verification for free without additional training, which enables both rejecting false positive matches and allows for completing more loop closures. Finally, we conduct a suite of experiments which includes showing VGGT-SLAM 2.0 can easily be adapted for open-set object detection and demonstrating real time performance while running online onboard a ground robot using a Jetson Thor. We also test in environments ranging from cluttered indoor apartments and office scenes to a 4,200 square foot barn, and we also demonstrate VGGT-SLAM 2.0 achieves the highest accuracy on the TUM dataset with about 23 percent less pose error than VGGT-SLAM. Code will be released upon publication.

Figures (9)

• Figure 1: A VGGT-SLAM 2.0 map of an apartment scene showing a living room, kitchen, bedroom, 2 bathrooms, and closet using uncalibrated RGB images collected with an iPhone. After the map is constructed, it is easy to use it for retrieval of open-set objects. Here, eight example open-set object queries are included showing the resulting 3D bounding box produced by VGGT-SLAM 2.0 given the provided text query.
• Figure 2: Office loop dataset from Maggio25neurips-VGGT-SLAM. Left: VGGT-SLAM Maggio25neurips-VGGT-SLAM before detected loop closure showing high-dimensional drift. Middle: VGGT-SLAM 2.0 before detected loop closure showing substantially reduced drift. Right: VGGT-SLAM 2.0 after detected loop closure
• Figure 3: Factor graph structure of VGGT-SLAM 2.0 showing edges inside and between submaps where all keyframes are nodes and loop closure submaps consist of two frames. Nodes connected with an inner submap edge such as $H_3$ and $H_4$ are an example of overlapping nodes (meaning the camera image associated with these nodes is identical). Here, for the loop closure submap which is made up of nodes $H_{14}$ and $H_{15}$, $H_{12}$ represents the node of a queried frame which was matched with the node $H_3$ of the retrieved frame.
• Figure 4: Attention matrix of query tokens for pairs of images with respect to a selected key token (identified with a black star) using tokens from layers 21, 22, and 23. Layer 22 shows a clear spotlight of attention between corresponding parts of the image pairs.
• Figure 5: Examples using layer 22 to verify a match between queried and retrieved frames. We visualize attention maps between a selected key token and all query tokens revealing high attention for matching frames and low attention for non-matching frames. This is captured for all pairs of key and query tokens using our match score produced with \ref{['eq:match_score']}.