KV-Tracker: Real-Time Pose Tracking with Transformers

TL;DR

KV-Tracker tackles real-time 3D pose tracking and online reconstruction from monocular RGB streams by leveraging multi-view transformer networks while avoiding the prohibitive cost of reprocessing all frames online. It builds a KV-cache from a set of keyframes during a mapping phase and relocalizes the current frame against this cached representation during tracking, reducing computational complexity and achieving real-time performance (up to about 27 FPS) with a reported speedup of around 15x over full re-computation. The approach supports both scene-level tracking and zero-shot object tracking without depth data or CAD priors and is designed to be compatible with off-the-shelf multi-view models without retraining. Extensive experiments across TUM RGB-D, 7-Scenes, Arctic, and OnePose datasets demonstrate strong accuracy and real-time capabilities, with linear memory growth tied to the number of keyframes and potential applicability to other attention-based reconstruction models. Overall, KV-Tracker provides a practical, training-free pathway to online multi-view 3D tracking and reconstruction in constrained settings.

Abstract

Multi-view 3D geometry networks offer a powerful prior but are prohibitively slow for real-time applications. We propose a novel way to adapt them for online use, enabling real-time 6-DoF pose tracking and online reconstruction of objects and scenes from monocular RGB videos. Our method rapidly selects and manages a set of images as keyframes to map a scene or object via $π^3$ with full bidirectional attention. We then cache the global self-attention block's key-value (KV) pairs and use them as the sole scene representation for online tracking. This allows for up to $15\times$ speedup during inference without the fear of drift or catastrophic forgetting. Our caching strategy is model-agnostic and can be applied to other off-the-shelf multi-view networks without retraining. We demonstrate KV-Tracker on both scene-level tracking and the more challenging task of on-the-fly object tracking and reconstruction without depth measurements or object priors. Experiments on the TUM RGB-D, 7-Scenes, Arctic and OnePose datasets show the strong performance of our system while maintaining high frame-rates up to ${\sim}27$ FPS.

Figures (6)

• Figure 1: Real-time object tracking and online reconstruction using an RGB images from a camera scanning a telephone booth. Red frustums indicate keyframes used for mapping; blue frustums indicate tracking frames processed at ${\sim}27$ FPS. The geometry shows the global reconstruction obtained by fusing point maps from keyframes.
• Figure 2: This is an overview of our real-time tracking and online reconstruction method. Our system is decomposed into 2 interleaved steps mapping and tracking. During the mapping stage a set of keyframes $KF_{1:B}$ are used to generate a KV-cache. During the tracking the latest frame $I_t$ tracked using the latest available cached key-value pairs. The KV-cache is refreshed with new keyframe insertions. During object level mode, segmentation masks can be used, along with our keyframing system.
• Figure 3: Visualisation of the full bidirectional global self-attention used during mapping to generate the KV-cache (left) vs. cross-attention with KV-cache + self-attention used for tracking of live frames (right).
• Figure 4: Tracking and reconstruction demo on a moving car, captured using a phone in the wild.
• Figure 5: Sample trajectory and reconstruction results from the object tracking evaluation datasets. The Arctic dataset samples are run with $308\times308$ images. While the OnePose dataset and OnePose Low Texture dataset samples are run on $518\times518$.