Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks

TL;DR

This work tackles modality-missing challenges in RGBT tracking by introducing Invertible Prompt Learning (IPL), which generates content-preserving prompts for the missing modality from the available one and enforces bidirectional reconstruction to mitigate semantic distortion. A two-stage training paradigm first solidifies a robust baseline tracker and then specializes the invertible prompters with task-alignment and bidirectional losses, enabling effective operation only when modality is missing. The authors also construct three high-quality modality-missing benchmarks (RGBT234-Miss, LasHeR245-Miss, VTUAV176-Miss) with hierarchical missing patterns and ratios to reflect real-world scenarios. Extensive experiments demonstrate that IPL consistently outperforms state-of-the-art trackers in both modality-complete and modality-missing settings, validating its robustness and practical value for multimodal tracking in drone and surveillance contexts.

Abstract

Current RGBT tracking research relies on the complete multi-modal input, but modal information might miss due to some factors such as thermal sensor self-calibration and data transmission error, called modality-missing challenge in this work. To address this challenge, we propose a novel invertible prompt learning approach, which integrates the content-preserving prompts into a well-trained tracking model to adapt to various modality-missing scenarios, for robust RGBT tracking. Given one modality-missing scenario, we propose to utilize the available modality to generate the prompt of the missing modality to adapt to RGBT tracking model. However, the cross-modality gap between available and missing modalities usually causes semantic distortion and information loss in prompt generation. To handle this issue, we design the invertible prompter by incorporating the full reconstruction of the input available modality from the generated prompt. To provide a comprehensive evaluation platform, we construct several high-quality benchmark datasets, in which various modality-missing scenarios are considered to simulate real-world challenges. Extensive experiments on three modality-missing benchmark datasets show that our method achieves significant performance improvements compared with state-of-the-art methods. We have released the code and simulation datasets at: \href{https://github.com/Alexadlu/Modality-missing-RGBT-Tracking.git}{https://github.com/Alexadlu/Modality-missing-RGBT-Tracking.git}.

Figures (15)

• Figure 1: Illustration of modality-missing factors and evaluation results of state-of-the-art methods on the RGBT234-Miss dataset.
• Figure 2: Three feature extractors in RGBT tracking models.
• Figure 3: Modality-missing RGBT tracking framework with the invertible prompt learning.
• Figure 4: Visualization of feature maps between the proposed Invertible Prompter (IPer) and popular adapter LoRA hu2021lora.
• Figure 5: Illustration of Invertible Prompter (IPer) block.