Beyond Visual Cues: Synchronously Exploring Target-Centric Semantics for Vision-Language Tracking

TL;DR

This paper tackles robust Vision-Language tracking by integrating target-centric semantics through a Synchronous Learning Backbone (SLB) that enables simultaneous, cross-modal feature extraction and interaction. It introduces the Target Enhance Module (TEM) and Semantic Aware Module (SAM) to progressively fuse visual and textual information, and a Dense Matching loss to directly optimize multi-modal representations. The proposed SATracker achieves state-of-the-art results on VL benchmarks such as TNL2K and OTB99, and excels on LaSOT, with ablations confirming the contributions of TEM, SAM, and DM. This work demonstrates the practical impact of synchronized vision-language learning for resilient tracking in complex, noisy scenes.

Abstract

Single object tracking aims to locate one specific target in video sequences, given its initial state. Classical trackers rely solely on visual cues, restricting their ability to handle challenges such as appearance variations, ambiguity, and distractions. Hence, Vision-Language (VL) tracking has emerged as a promising approach, incorporating language descriptions to directly provide high-level semantics and enhance tracking performance. However, current VL trackers have not fully exploited the power of VL learning, as they suffer from limitations such as heavily relying on off-the-shelf backbones for feature extraction, ineffective VL fusion designs, and the absence of VL-related loss functions. Consequently, we present a novel tracker that progressively explores target-centric semantics for VL tracking. Specifically, we propose the first Synchronous Learning Backbone (SLB) for VL tracking, which consists of two novel modules: the Target Enhance Module (TEM) and the Semantic Aware Module (SAM). These modules enable the tracker to perceive target-related semantics and comprehend the context of both visual and textual modalities at the same pace, facilitating VL feature extraction and fusion at different semantic levels. Moreover, we devise the dense matching loss to further strengthen multi-modal representation learning. Extensive experiments on VL tracking datasets demonstrate the superiority and effectiveness of our methods.

Figures (6)

• Figure 1: The pipeline of existing VL trackers (a) and ours (b). We boost the performance and improve the tracking pipeline via synchronous Vision-Language feature extraction and interaction.
• Figure 2: Architecture of the proposed tracking framework. Both the template and search regions, along with the language description, are tokenized into sequences, which are subsequently sent into the synchronous learning backbone. Via the incorporation of TEM and SAM, the backbone network progressively facilitates synchronous feature extraction and interaction between the visual and textual modalities. Finally, the semantic-guided search feature is utilized for target localization through a plain corner prediction head.
• Figure 3: An elaborate illustration of the Target Enhance Module, which efficiently performs self-attention and asymmetrical cross-attention to enhance target-relevant features.
• Figure 4: The detailed computation process of the Semantic Aware Module (SAM), comprising the visual stream (lower side) and textual stream (upper side).
• Figure 5: The impact of the starting stage of SAM.