CLDTracker: A Comprehensive Language Description for Visual Tracking

TL;DR

CLDTracker tackles open-vocabulary visual tracking by introducing a Comprehensive Language Description framework that jointly leverages rich textual descriptions and temporal language adaptation. It builds a large, structured Bag of Textual Descriptions ($B_t$) through CLIP-based class/attribute matching, GPT-4V-generated scene captions, and a semantic-context enrichment pipeline, then uses a Prompt Adapter to select instance-relevant prompts. A Temporal Text Feature Update Mechanism ($TTFUM$) updates textual cues over time, enabling robust tracking under appearance changes, occlusions, and clutter by aligning the textual bag with visual features via a convolution-based image-bag correlation. Empirical results on six standard VOT benchmarks show strong performance gains and competitive generalization, with detailed ablations validating the contribution of each component and demonstrating practical real-time efficiency. The work suggests a promising direction for open-vocabulary tracking by tightly integrating rich, temporally adaptive language representations with visual localization, while also highlighting areas for future improvement in long-term occlusion handling and prompt quality control.

Abstract

VOT remains a fundamental yet challenging task in computer vision due to dynamic appearance changes, occlusions, and background clutter. Traditional trackers, relying primarily on visual cues, often struggle in such complex scenarios. Recent advancements in VLMs have shown promise in semantic understanding for tasks like open-vocabulary detection and image captioning, suggesting their potential for VOT. However, the direct application of VLMs to VOT is hindered by critical limitations: the absence of a rich and comprehensive textual representation that semantically captures the target object's nuances, limiting the effective use of language information; inefficient fusion mechanisms that fail to optimally integrate visual and textual features, preventing a holistic understanding of the target; and a lack of temporal modeling of the target's evolving appearance in the language domain, leading to a disconnect between the initial description and the object's subsequent visual changes. To bridge these gaps and unlock the full potential of VLMs for VOT, we propose CLDTracker, a novel Comprehensive Language Description framework for robust visual Tracking. Our tracker introduces a dual-branch architecture consisting of a textual and a visual branch. In the textual branch, we construct a rich bag of textual descriptions derived by harnessing the powerful VLMs such as CLIP and GPT-4V, enriched with semantic and contextual cues to address the lack of rich textual representation. Experiments on six standard VOT benchmarks demonstrate that CLDTracker achieves SOTA performance, validating the effectiveness of leveraging robust and temporally-adaptive vision-language representations for tracking. Code and models are publicly available at: https://github.com/HamadYA/CLDTracker

Figures (9)

• Figure 1: VOT performance comparison in terms of Area Under the Curve (AUC) score between the CiteTracker citetracker containing limited textual descriptions and our proposed CLDTracker containing comprehensive textual descriptions of the target object. CLDTracker shows significant performance improvements on six VOT benchmark datasets including LaSOT lasot, LaSOTextlasotext, TNL2K tnl2k, GOT-10k got10k, TrackingNet trackingnet, and OTB99-Lang lang_tracker.
• Figure 2: Comparison of the visual-textual alignment process in the VLM-based trackers. (a) show the traditional visual-textual alignment in the SOTA VLM-based trackers jointnltcitetracker and (b) shows our proposed CLDTracker containing a comprehensive bag of textual descriptions ($B_t$) which is then aligned with the visual representation and dynamic text generated from our Temporal Text Feature Update Mechanism (TTFUM).
• Figure 3: Schematic illustration of our proposed Comprehensive Language Description framework for Visual Tracking (CLDTracker). Our tracker comprises of two main branches including textual and visual. (A) shows our textual branch in which we first extract visual features from exemplar and search images using CLIP image encoder clip (Steps (a) & (b)), then we construct a $B_t$ illustrated in Fig. \ref{['fig:Bag']} followed by the Prompt Adapter component and TTFUM in steps (c) & (d). (B) shows our vision branch in which we extract the visual features and then estimate the correlation with textual representation for efficient visual tracking.
• Figure 4: System diagram illustrating the process of building a $B_t$. Steps (a) shows our class and corresponding attributes-based dictionaries. Step (b) shows the first frame of sequence containing target object tightly coupled with a bounding box. Steps (c)-(f) outline the process of identifying best class and attributes based on target object shown in step (b). Steps (g) & (h) show the process of image-to-text descriptions using GPT-4V. Steps (i) & (j) show the extension of the textual description by incorporating semantic and contextual information ($S\&C$), and step (k) shows an entire $B_t$ containing overall descriptions.
• Figure 5: AUC scores of different attributes on (a) LaSOT, and (b) LaSOText. Best viewed in color.