Collaborating Vision, Depth, and Thermal Signals for Multi-Modal Tracking: Dataset and Algorithm

TL;DR

This work tackles robust object tracking in challenging environments by introducing tri-modal RGB-D-T tracking. It presents RGBDT500, a synchronized dataset with 400 training and 100 test sequences across 66 classes, and a baseline tracker RDTTrack that fuses Depth and TIR via an orthogonal projection module and then uses multi-modal prompts to adapt a frozen OSTrack model. The training optimizes a composite loss $L = L_{\text{CLS}} + \lambda_{\text{GIoU}} L_{\text{GIoU}} + \lambda_{L_{1}} L_{1}$, while keeping the RGB branch fixed. Experiments show that RDTTrack outperforms uni- and dual-modal trackers with a runtime of 76.6 FPS and only 0.86M trainable parameters, demonstrating effective tri-modal fusion and practical efficiency.

Abstract

Existing multi-modal object tracking approaches primarily focus on dual-modal paradigms, such as RGB-Depth or RGB-Thermal, yet remain challenged in complex scenarios due to limited input modalities. To address this gap, this work introduces a novel multi-modal tracking task that leverages three complementary modalities, including visible RGB, Depth (D), and Thermal Infrared (TIR), aiming to enhance robustness in complex scenarios. To support this task, we construct a new multi-modal tracking dataset, coined RGBDT500, which consists of 500 videos with synchronised frames across the three modalities. Each frame provides spatially aligned RGB, depth, and thermal infrared images with precise object bounding box annotations. Furthermore, we propose a novel multi-modal tracker, dubbed RDTTrack. RDTTrack integrates tri-modal information for robust tracking by leveraging a pretrained RGB-only tracking model and prompt learning techniques. In specific, RDTTrack fuses thermal infrared and depth modalities under a proposed orthogonal projection constraint, then integrates them with RGB signals as prompts for the pre-trained foundation tracking model, effectively harmonising tri-modal complementary cues. The experimental results demonstrate the effectiveness and advantages of the proposed method, showing significant improvements over existing dual-modal approaches in terms of tracking accuracy and robustness in complex scenarios. The dataset and source code are publicly available at https://xuefeng-zhu5.github.io/RGBDT500.

Figures (6)

• Figure 1: Representative tri-modal samples of RGBDT500. In each sample, at least one modality is affected by specific challenges, highlighting the need for more general multi-modal fusion.
• Figure 2: The object category distribution of the RGBDT500 test set.
• Figure 3: An overview of the pipeline and architecture of RDTTrack. (a) illustrates the overall tracking pipeline; (b) depicts the detailed structure of the Depth-TIR fusion module; and (c) presents the architecture of the prompt learning block.
• Figure 4: The precision plots and success plots of trackers on RGBDT500.
• Figure 5: Some samples from the RGBDT500 dataset. The targeted object are highlighted by green bounding boxes.