Robust Tracking via Mamba-based Context-aware Token Learning

TL;DR

TemTrack addresses the challenge of robust visual tracking under limited computational budgets by separating temporal information learning from appearance modeling. It introduces a per-frame track token and a mamba-based Temporal Module that operates over a sliding window to capture appearance changes and motion trends, reducing the need to process multiple images. The method achieves competitive or state-of-the-art results across multiple benchmarks (LaSOT, GOT-10k, TrackingNet, UAV123, TNL2K) while maintaining real-time speeds and lower FLOPs compared to image-heavy temporal approaches. This approach offers a practical path toward efficient, robust tracking suitable for real-world deployment in resource-constrained environments.

Abstract

How to make a good trade-off between performance and computational cost is crucial for a tracker. However, current famous methods typically focus on complicated and time-consuming learning that combining temporal and appearance information by input more and more images (or features). Consequently, these methods not only increase the model's computational source and learning burden but also introduce much useless and potentially interfering information. To alleviate the above issues, we propose a simple yet robust tracker that separates temporal information learning from appearance modeling and extracts temporal relations from a set of representative tokens rather than several images (or features). Specifically, we introduce one track token for each frame to collect the target's appearance information in the backbone. Then, we design a mamba-based Temporal Module for track tokens to be aware of context by interacting with other track tokens within a sliding window. This module consists of a mamba layer with autoregressive characteristic and a cross-attention layer with strong global perception ability, ensuring sufficient interaction for track tokens to perceive the appearance changes and movement trends of the target. Finally, track tokens serve as a guidance to adjust the appearance feature for the final prediction in the head. Experiments show our method is effective and achieves competitive performance on multiple benchmarks at a real-time speed. Code and trained models will be available at https://github.com/GXNU-ZhongLab/TemTrack.

Figures (6)

• Figure 1: Comparison of AUC, Params, and FlOPs of recent SOTA trackers. The trackers that lower Params and higher AUC is closer to top-left corner. The size of a circles represents the tracker’s FLOPs.
• Figure 2: Overview of the proposed tracker TemTrack. The tracker's workflow is depicted from left to right, including feature extraction & interaction, temporal information modeling, and the final head stage. First, we add a track token $\bm F_t$ concatenating with template and search tokens to gather the target's appearance in the backbone. Furthermore, we develop a Temporal Module to associate track tokens to dig temporal information. Finally, the track tokens guide the adjustment of the search features to achieve more accurate predictions in the head network.
• Figure 3: The schematic diagram of the Mamba_Cross. The $\bm{F}_t^N$ and $\bm{H}_t$ indicate track token and the hidden state at the $t$ frame. After this module, the track token $\bm{F}_t^{N"}$ gathers the appearance of the previous frames within a sliding window.
• Figure 4: AUC scores of difference attributes on LaSOTlasot. Best viewed in color.
• Figure 5: Success plots of one-pass evaluation (OPE) about fast motion and full occlusion challenges on LaSOT. Best viewed in color and zooming in.