Keyframe-Based Feed-Forward Visual Odometry

TL;DR

This work tackles the inefficiency of end-to-end visual foundation-model visual odometry by introducing a reinforcement-learning–based keyframe policy that adaptively filters input frames within a sliding-window VO pipeline built on VGGT. The method trains on synthetic data (TartanAir) and demonstrates strong generalization to real-world datasets (EuRoC, TUM-RGBD, KITTI) while maintaining competitive accuracy with minimal runtime overhead. By explicitly maintaining and selecting keyframes, the approach improves local estimation and robustness without requiring explicit geometric heuristics or post-processing. The work highlights the potential of data-driven memory management for scalable, feed-forward VO and points to future directions toward fully feed-forward SLAM with loop closure.

Abstract

The emergence of visual foundation models has revolutionized visual odometry~(VO) and SLAM, enabling pose estimation and dense reconstruction within a single feed-forward network. However, unlike traditional pipelines that leverage keyframe methods to enhance efficiency and accuracy, current foundation model based methods, such as VGGT-Long, typically process raw image sequences indiscriminately. This leads to computational redundancy and degraded performance caused by low inter-frame parallax, which provides limited contextual stereo information. Integrating traditional geometric heuristics into these methods is non-trivial, as their performance depends on high-dimensional latent representations rather than explicit geometric metrics. To bridge this gap, we propose a novel keyframe-based feed-forward VO. Instead of relying on hand-crafted rules, our approach employs reinforcement learning to derive an adaptive keyframe policy in a data-driven manner, aligning selection with the intrinsic characteristics of the underlying foundation model. We train our agent on TartanAir dataset and conduct extensive evaluations across several real-world datasets. Experimental results demonstrate that the proposed method achieves consistent and substantial improvements over state-of-the-art feed-forward VO methods.

Figures (5)

• Figure 1: Method Overview: We propose a keyframe-based feed-forward VO with VGGT wang2025vggt as backbone and a RL-based keyframe method. The observation consists of the mean of CLS tokens extracted by the ViT encoder from all frames within the keyframe window, together with the relative pose changes. Based on this observation, the agent decides whether to retain the new input frame as a keyframe or not. After executing the action, the predicted poses are aligned with the ground-truth poses then be evaluated by the reward function, the result will be used to update the policy. During inference, the poses of all keyframe chunks are aligned by using the anchor frame, yielding the final pose estimation for the entire sequence. For clarity, some implementation details are omitted in the diagram to emphasize the overall framework and core workflow.
• Figure 2: Visualized trajectory of our method, VGGT-Long, VGGT-SLAM and InfiniteVGGT on MH_02 from EuRoC. The ground-truth trajectory is shown in red, while the estimated trajectories are shown in blue.
• Figure 3: Visualized trajectory of our method, VGGT-Long, VGGT-SLAM and InfiniteVGGT on fr1-xyz from TUM-RGBD. The ground-truth trajectory is shown in red, while the estimated trajectories are shown in blue.
• Figure 4: Visualized trajectory of our method, VGGT-Long, VGGT-SLAM and InfiniteVGGT on 02 from KITTI. The ground-truth trajectory is shown in red, while the estimated trajectories are shown in blue.
• Figure 5: Visualized trajectory of VGGT-SW and VGGT-LK on fr1-xyz from TUM-RGBD. The ground-truth trajectory is shown in red, while the estimated trajectories are shown in blue.