Optimized Information Flow for Transformer Tracking

TL;DR

This work identifies and addresses the weaknesses of free bidirectional information flow in one-stream Transformer trackers. It introduces OIFTrack, which optimizes token interactions by masking early flows and partitioning search tokens in deeper layers, while incorporating dynamic target and background cues to capture appearance changes and surrounding context. The approach yields state-of-the-art results on GOT-10k and strong performance across TrackingNet, LaSOT, and UAV123, supported by extensive ablations that validate the design choices. Overall, the method advances robust, discriminative tracking in challenging scenes with practical computation, highlighting the value of controlled information flow in vision transformers for tracking tasks.

Abstract

One-stream Transformer trackers have shown outstanding performance in challenging benchmark datasets over the last three years, as they enable interaction between the target template and search region tokens to extract target-oriented features with mutual guidance. Previous approaches allow free bidirectional information flow between template and search tokens without investigating their influence on the tracker's discriminative capability. In this study, we conducted a detailed study on the information flow of the tokens and based on the findings, we propose a novel Optimized Information Flow Tracking (OIFTrack) framework to enhance the discriminative capability of the tracker. The proposed OIFTrack blocks the interaction from all search tokens to target template tokens in early encoder layers, as the large number of non-target tokens in the search region diminishes the importance of target-specific features. In the deeper encoder layers of the proposed tracker, search tokens are partitioned into target search tokens and non-target search tokens, allowing bidirectional flow from target search tokens to template tokens to capture the appearance changes of the target. In addition, since the proposed tracker incorporates dynamic background cues, distractor objects are successfully avoided by capturing the surrounding information of the target. The OIFTrack demonstrated outstanding performance in challenging benchmarks, particularly excelling in the one-shot tracking benchmark GOT-10k, achieving an average overlap of 74.6\%. The code, models, and results of this work are available at \url{https://github.com/JananiKugaa/OIFTrack}

Figures (8)

• Figure 1: Comparison of the information flow of tokens between previous one-stream Transformer trackers (top) and proposed approach (bottom). Attention between all target and search region tokens are computed in previous one-stream Transformer trackers, while the proposed approach selectively blocks attention to enhance the discriminative ability of the tracker.
• Figure 2: Illustration of the proposed temporal cue utilization mechanism. We have extracted the temporal cues of the target as dynamic target patches and the surrounding cues as dynamic background patches. Since similar objects from the surrounding region are also identified as dynamic background patches, the proposed tracker can easily avoid distractor objects and accurately locate the target in the search region.
• Figure 3: (a): The overall tracking framework of the proposed tracker. The diagram illustrates the proposed attention feature extraction mechanism only in the last few encoder layers, and values are omitted for clarity. (b): Illustration of the optimized information flow between tokens in early encoder layers and deeper encoder layers.
• Figure 4: Top: Information flow diagram for the baseline model and model A, which blocks the interaction from search tokens ($E_{X}$) to target template tokens ($E_{Z}$). Bottom: Corresponding heatmap illustration of the search region for a tracking sequence.
• Figure 5: Top: Information flow diagrams for Model B, which includes dynamic target ($E_{DT}$) cues, and for Model C, which includes both dynamic target and background ($E_{DB}$) cues. Bottom: Corresponding heatmap visualization of the search region based on the classification scores.