AViTMP: A Tracking-Specific Transformer for Single-Branch Visual Tracking

TL;DR

AViTMP addresses the mismatch between vanilla Vision Transformer backbones and the specialized requirements of visual tracking, particularly in autonomous driving contexts. It introduces a tracking-tailored Adaptive ViT Encoder (AViT-Enc) with a Joint State Embedding and a lightweight Adaptor to inject image priors directly into a single-branch pipeline, paired with a Transformer Model Predictor (TMP) that yields target-model weights for precise localization. A novel online inference pipeline, CycleTrack, enforces temporal cycle consistency to improve robustness against distractors, complemented by a dual-frames update strategy that adapts templates without extra training costs. Across eight large benchmarks, AViTMP achieves state-of-the-art performance, especially in long-term tracking and robustness, while maintaining real-time efficiency (≈40 FPS), demonstrating a strong practical impact for reliable single-branch visual tracking in real-world systems.

Abstract

Visual object tracking is a fundamental component of transportation systems, especially for intelligent driving. Despite achieving state-of-the-art performance in visual tracking, recent single-branch trackers tend to overlook the weak prior assumptions associated with the Vision Transformer (ViT) encoder and inference pipeline in visual tracking. Moreover, the effectiveness of discriminative trackers remains constrained due to the adoption of the dual-branch pipeline. To tackle the inferior effectiveness of vanilla ViT, we propose an Adaptive ViT Model Prediction tracker (AViTMP) to design a customised tracking method. This method bridges the single-branch network with discriminative models for the first time. Specifically, in the proposed encoder AViT encoder, we introduce a tracking-tailored Adaptor module for vanilla ViT and a joint target state embedding to enrich the target-prior embedding paradigm. Then, we combine the AViT encoder with a discriminative transformer-specific model predictor to predict the accurate location. Furthermore, to mitigate the limitations of conventional inference practice, we present a novel inference pipeline called CycleTrack, which bolsters the tracking robustness in the presence of distractors via bidirectional cycle tracking verification. In the experiments, we evaluated AViTMP on eight tracking benchmarks for a comprehensive assessment, including LaSOT, LaSOTExtSub, AVisT, etc. The experimental results unequivocally establish that, under fair comparison, AViTMP achieves state-of-the-art performance, especially in terms of long-term tracking and robustness. The source code will be released at https://github.com/Tchuanm/AViTMP.

Figures (8)

• Figure 1: Performance comparison of vanilla ViT vs. our tracking-tailored AViT-Enc on the LaSOTExtSub Fan_2020_IJCV_Lasot_ext and LaSOT lasot datasets. We report success rate (SR), precision rate (PR), and normalized precision rate (NPR). Our designed AViT improves performance by around 5% on each metric on both datasets compared with vanilla ViT.
• Figure 2: Comparison among trackers that employ (a) single-branch paradigm, (b) discriminative transformer model, and (c) our proposed single-branch discriminative predictor AViTMP. Our model harnesses the strengths of the first two paradigms by incorporating the proposed encoder-decoder design. In this way, AViTMP integrates the powerful feature extraction capability from single-branch trackers and target prediction ability from discriminate methods into one pipeline.
• Figure 3: Overview of our proposed AViTMP architecture, including the AViT-Enc encoder and the Transformer-specific Model Pedictor (TMP) composing of a DF-Dec decoder and a final target prediction module. In AViT-Enc, training and test frames are contacted together and jointly encoded with the target prior information embedding. In DF-Dec, the encoded and adapted features are decoded and then densely integrated layer-wise to generate the model weights. Finally, the target model employs the adapted feature and model weights to predict the target location.
• Figure 4: Overview of proposed online inference strategies. (1) CycleTrack (top) consists of two different tracking processes, named forward and backward inference; (2) Dual-Frames Update (bottom) update two templates over time and conditions.
• Figure 5: Spatial consistent and inconsistent prediction along the temporal adjacent frames. The solid line represents the forward track process, while the dotted line indicates the backward track process to verify the prediction of top-2 results.