Revisiting RGBT Tracking Benchmarks from the Perspective of Modality Validity: A New Benchmark, Problem, and Solution

TL;DR

This work tackles the mismatch between existing RGB-T tracking benchmarks and real-world multi-modal warranting (MMW) conditions by introducing MV-RGBT, a diverse benchmark focused on modality validity captured exclusively in challenging RGB and/or TIR scenarios. It proposes MoETrack, a mixture-of-experts tracker with RGB, TIR, and RGBT heads and a modality switcher that selects the most reliable expert per frame, thereby addressing the 'when to fuse' problem. MV-RGBT enables compositional analysis by dividing data into MV-RGBT-RGB and MV-RGBT-TIR subsets, revealing modality validity patterns and guiding robust fusion strategies. Across MV-RGBT and other benchmarks (GTOT, RGBT234, LasHeR), MoETrack achieves state-of-the-art performance, while experiments demonstrate that dense fusion is not always beneficial in MMW contexts. The work thus advances practical RGBT tracking by promoting modality-balanced evaluation and adaptive fusion, with broader implications for multi-modal perception tasks.

Abstract

RGBT tracking draws increasing attention because its robustness in multi-modal warranting (MMW) scenarios, such as nighttime and adverse weather conditions, where relying on a single sensing modality fails to ensure stable tracking results. However, existing benchmarks predominantly contain videos collected in common scenarios where both RGB and thermal infrared (TIR) information are of sufficient quality. This weakens the representativeness of existing benchmarks in severe imaging conditions, leading to tracking failures in MMW scenarios. To bridge this gap, we present a new benchmark considering the modality validity, MV-RGBT, captured specifically from MMW scenarios where either RGB (extreme illumination) or TIR (thermal truncation) modality is invalid. Hence, it is further divided into two subsets according to the valid modality, offering a new compositional perspective for evaluation and providing valuable insights for future designs. Moreover, MV-RGBT is the most diverse benchmark of its kind, featuring 36 different object categories captured across 19 distinct scenes. Furthermore, considering severe imaging conditions in MMW scenarios, a new problem is posed in RGBT tracking, named `when to fuse', to stimulate the development of fusion strategies for such scenarios. To facilitate its discussion, we propose a new solution with a mixture of experts, named MoETrack, where each expert generates independent tracking results along with a confidence score. Extensive results demonstrate the significant potential of MV-RGBT in advancing RGBT tracking and elicit the conclusion that fusion is not always beneficial, especially in MMW scenarios. Besides, MoETrack achieves state-of-the-art results on several benchmarks, including MV-RGBT, GTOT, and LasHeR. Github: https://github.com/Zhangyong-Tang/MVRGBT.

Figures (12)

• Figure 1: The proposed benchmark is inspired by the observed inconsistency between the data in existing benchmarks and the imaging conditions motivating RGBT tracking. $\rm {Re_{RGB}}$, $\rm{Re_{TIR}}$, and $\rm{Re_{RGBT}}$ represent the reliabilities of predictions from RGB, TIR, and the fused (RGBT) experts, respectively. On the right side, the statistics on existing datasets are provided and the entire list will be available at the project page.
• Figure 2: (a) Object classes and scenes of the proposed MV-RGBT; (b) Illustration of the key point-based alignment method.
• Figure 3: Differences between the existing datasets and the proposed MV-RGBT. (a) and (b) shows the image-level differences through histogram. (c) depicts the differences of data distributions though T-SNE.
• Figure 4: Pipeline of MoETrack. Based on ViT-B-256, MoETrack employs a mixture of experts. During training, the gradients of multiple experts are computed separately, resulting in a jointly optimised backbone. In the test stage, a modality switcher is utilised, only activating the modality with best-evaluated reliability.
• Figure 5: Reasons for posing the new problem 'when to fuse' with samples from MV-RGBT ($ET\_Person\_SkatingRink$ and $ER\_Bar\_Bedroom0$).