GLAD: Generative Language-Assisted Visual Tracking for Low-Semantic Templates

TL;DR

GLAD tackles vision-language tracking under low-semantic templates by introducing a diffusion-based generative fusion framework. It fuses text and template in a latent diffusion space via a Generative Diffusion Module to restore semantic richness, then refines multimodal features through cascaded Multi-Modal Decoders before a CenterNet-style head localizes the target. The approach, aided by a Latent Consistency Model for efficiency, achieves state-of-the-art results on LaSOT, LaSOT_ext, and TNL2K, while maintaining high inference speed (~44 FPS) and robustness to blur, occlusion, and distribution shifts. The work provides a principled path for language-guided template enhancement and multimodal fusion, with substantial practical impact for real-time tracking in noisy visual scenes.

Abstract

Vision-language tracking has gained increasing attention in many scenarios. This task simultaneously deals with visual and linguistic information to localize objects in videos. Despite its growing utility, the development of vision-language tracking methods remains in its early stage. Current vision-language trackers usually employ Transformer architectures for interactive integration of template, search, and text features. However, persistent challenges about low-semantic images including prevalent image blurriness, low resolution and so on, may compromise model performance through degraded cross-modal understanding. To solve this problem, language assistance is usually used to deal with the obstacles posed by low-semantic images. However, due to the existing gap between current textual and visual features, direct concatenation and fusion of these features may have limited effectiveness. To address these challenges, we introduce a pioneering Generative Language-AssisteD tracking model, GLAD, which utilizes diffusion models for the generative multi-modal fusion of text description and template image to bolster compatibility between language and image and enhance template image semantic information. Our approach demonstrates notable improvements over the existing fusion paradigms. Blurry and semantically ambiguous template images can be restored to improve multi-modal features in the generative fusion paradigm. Experiments show that our method establishes a new state-of-the-art on multiple benchmarks and achieves an impressive inference speed. The code and models will be released at: https://github.com/Confetti-lxy/GLAD

Figures (7)

• Figure 1: Paradigms for Vision-Language trackers. The top section shows the inputs, including text, search image and template image. (a) Traditional paradigm on the left directly concatenations different features and conducts a simple unified fusion. (b) Our generative paradigm on the right employs diffusion models for restoration of suboptimal template images, and interactions with original images are then conducted during generative fusion.
• Figure 2: Radar plot of AUC scores for LaSOT attributes, including Motion Blur, Camera Motion, Deformation, Illumination Variation, Out-of-View, Scale Variation, and Background Clutter. The figure illustrates the tracking performance of different models under diverse challenging conditions. A larger enclosed area indicates better overall tracking performance across all attribute types.
• Figure 3: Pipeline of the proposed GLAD model. (a) Firstly, the natural language and the template image will be generatively fused using the Generative Diffusion Module. Secondly, the generated diffusion features are passed through the cascaded Multi-Modal Decoder to enhance semantic information. Search features are concatenated with template features for decoding modules under the guidance of the fused features from pooling modules. (b) Details of Stable Diffusion U-Net. (c) Details of Multi-Modal Decoder when $N=3$.
• Figure 4: Overview of the proposed Attention Pooling Module. It aligns the spatial and dimensional discrepancies between U-Net and visual encoder features while preserving semantic richness for effective multi-modal fusion.
• Figure 5: Structure of the Feature Decoding Module. It enhances the semantic representation and discriminative ability of search features by leveraging multi-modal information from the Attention Pooling Module.