Exploring Reliable Spatiotemporal Dependencies for Efficient Visual Tracking

TL;DR

The paper tackles the efficiency-accuracy trade-off in visual tracking by introducing STDTrack, a lightweight transformer-based framework that densely samples video sequences during training to exploit rich spatiotemporal information. It couples a transformer encoder with a Multi-frame Information Fusion Module (MFIFM), a Spatiotemporal Token Maintainer (STM), and a multi-scale prediction head to propagate and reinforce temporal context across frames, while employing structural re-parameterization to maintain fast inference. Empirical results across six benchmarks show state-of-the-art performance among real-time trackers and competitive results with non-real-time methods, including strong GOT-10k results at high FPS. The work demonstrates that reliable spatiotemporal dependencies can be integrated into lightweight trackers to achieve robust, real-time tracking suitable for resource-constrained deployments.

Abstract

Recent advances in transformer-based lightweight object tracking have established new standards across benchmarks, leveraging the global receptive field and powerful feature extraction capabilities of attention mechanisms. Despite these achievements, existing methods universally employ sparse sampling during training--utilizing only one template and one search image per sequence--which fails to comprehensively explore spatiotemporal information in videos. This limitation constrains performance and cause the gap between lightweight and high-performance trackers. To bridge this divide while maintaining real-time efficiency, we propose STDTrack, a framework that pioneers the integration of reliable spatiotemporal dependencies into lightweight trackers. Our approach implements dense video sampling to maximize spatiotemporal information utilization. We introduce a temporally propagating spatiotemporal token to guide per-frame feature extraction. To ensure comprehensive target state representation, we disign the Multi-frame Information Fusion Module (MFIFM), which augments current dependencies using historical context. The MFIFM operates on features stored in our constructed Spatiotemporal Token Maintainer (STM), where a quality-based update mechanism ensures information reliability. Considering the scale variation among tracking targets, we develop a multi-scale prediction head to dynamically adapt to objects of different sizes. Extensive experiments demonstrate state-of-the-art results across six benchmarks. Notably, on GOT-10k, STDTrack rivals certain high-performance non-real-time trackers (e.g., MixFormer) while operating at 192 FPS(GPU) and 41 FPS(CPU).

Figures (6)

• Figure 1: Comparison of our STDTrack with others on the AVisT dataset AVisT on a CPU. The success score (AUC) (vertical axis) and speed (horizontal axis) are shown. Our tracker achieves substantial accuracy improvement over other state-of-the-art efficient trackers.
• Figure 2: The architecture of our STDTrack framework. It comprises four components: a transformer encoder for feature extraction, a Multi-frame Information Fusion Module (MFIFM) that enhances and refines spatiotemporal representations, a Spatiotemporal Token Maintainer (STM) for preserving high-quality temporal dependencies and an adaptive multi-scale prediction head.
• Figure 3: Architecture of MFIFM. This module fuses the spatiotemporal feature vectors ${\left\{ \textbf{F}_t \right\}}_{t=1}^{N}$ (summarizing target information per frame) into an augmented representation $F"_t$ through temporal propagation of historical dependencies.
• Figure 4: RepConv block design. (a) Standard Conv block in the center head OSTrack. (b) Our proposed RepConv block employing structural re-parameterization technique. During inference, this technique merges multi-branch convolutional layers into a single convolution operation.
• Figure 5: Qualitative comparison of our STDTrack and other SOTA trackers on LaSOT benchmark.