Modality-Guided Dynamic Graph Fusion and Temporal Diffusion for Self-Supervised RGB-T Tracking

TL;DR

GDSTrack tackles the challenge of self-supervised RGB-T tracking where pseudo-label noise hinders reliable multi-modal fusion. It introduces Modality-guided Dynamic Graph Fusion (MDGF) to adaptively fuse RGB and infrared features via a dynamic adjacency matrix, and Temporal Graph-Informed Diffusion (TGID) to denoise fused features using a diffusion model conditioned on prior frames. The combination yields robust tracking under weak supervision and distractor interference, achieving state-of-the-art results on GTOT, RGBT234, LasHeR, and VTUAV datasets. The work advances practical self-supervised RGB-T tracking by reducing annotation dependency and improving resilience to pseudo-label noise.

Abstract

To reduce the reliance on large-scale annotations, self-supervised RGB-T tracking approaches have garnered significant attention. However, the omission of the object region by erroneous pseudo-label or the introduction of background noise affects the efficiency of modality fusion, while pseudo-label noise triggered by similar object noise can further affect the tracking performance. In this paper, we propose GDSTrack, a novel approach that introduces dynamic graph fusion and temporal diffusion to address the above challenges in self-supervised RGB-T tracking. GDSTrack dynamically fuses the modalities of neighboring frames, treats them as distractor noise, and leverages the denoising capability of a generative model. Specifically, by constructing an adjacency matrix via an Adjacency Matrix Generator (AMG), the proposed Modality-guided Dynamic Graph Fusion (MDGF) module uses a dynamic adjacency matrix to guide graph attention, focusing on and fusing the object's coherent regions. Temporal Graph-Informed Diffusion (TGID) models MDGF features from neighboring frames as interference, and thus improving robustness against similar-object noise. Extensive experiments conducted on four public RGB-T tracking datasets demonstrate that GDSTrack outperforms the existing state-of-the-art methods. The source code is available at https://github.com/LiShenglana/GDSTrack.

Figures (6)

• Figure 1: (a) Existing RGB-T tracking methods rely heavily on a large number of ground truth annotations, making it challenging to handle larger-scale RGB-T datasets under supervised settings. (b) The discrepancy between the pseudo label and ground truth makes accurate object tracking difficult, causing performance degradation when used for the fully supervised training of sota trackers. (c) We leverage modality-guided dynamic graph fusion and temporal diffusion to address the issues of object irrelevant fusion and distractor noise caused during the training process supervised by pseudo-label. (d) Compared with the three trackers, our proposed method achieved the best performance under pseudo-label supervision.
• Figure 2: The pipeline of GDSTrack model. First, the encoder obtains the search frame features that interact with the template frame features. Subsequently, the MDGF module fuses the features of the two modalities using graph attention guided by dynamic adjacency matrix. Then the features obtained from the MDGF module are used as distractor noise and input for the TGID module to perform denoising and obtain the final tracking results. We use TGID to enhance the model's robustness to noise.
• Figure 3: The pipeline of AMG. The AMG module concatenates visible and infrared features along the sequence dimension, then computes cosine similarity and dot-product attention with themselves. After summation and threshold filtering, they guide the dot-product attention to generate similarity mask. Finally, SoftMax, TopK, and Symmetrize are applied to obtain the adjacency matrix.
• Figure 4: Attribute-based evaluation on VTUAV dataset compared against five self-supervised RGBT trackers. (a) Precision Rate with different attributes. (b) Success Rate with different attributes.
• Figure 5: PR and SR on RGBT234 dataset compared against other self-supervised RGBT trackers.