Breaking Shallow Limits: Task-Driven Pixel Fusion for Gap-free RGBT Tracking

TL;DR

The paper tackles the modality gap challenge in RGB-T tracking by showing that shallow, pixel-level fusion, while efficient, lacks discriminative power. It introduces Task-driven Pixel-level Fusion (TPF), featuring a lightweight Pixel-level Fusion Adapter (PFA) built on Mamba, and a two-stage progressive learning pipeline (MAD for cross-model knowledge distillation and DRF for decoupled task-relevant representation). A Nearest-Neighbor Dynamic Template Updating (NDTU) strategy further enhances robustness to appearance changes. Extensive experiments on four RGB-T benchmarks demonstrate state-of-the-art performance with real-time inference, highlighting the practical impact of task-directed pixel-level fusion for robust tracking.

Abstract

Current RGBT tracking methods often overlook the impact of fusion location on mitigating modality gap, which is key factor to effective tracking. Our analysis reveals that shallower fusion yields smaller distribution gap. However, the limited discriminative power of shallow networks hard to distinguish task-relevant information from noise, limiting the potential of pixel-level fusion. To break shallow limits, we propose a novel \textbf{T}ask-driven \textbf{P}ixel-level \textbf{F}usion network, named \textbf{TPF}, which unveils the power of pixel-level fusion in RGBT tracking through a progressive learning framework. In particular, we design a lightweight Pixel-level Fusion Adapter (PFA) that exploits Mamba's linear complexity to ensure real-time, low-latency RGBT tracking. To enhance the fusion capabilities of the PFA, our task-driven progressive learning framework first utilizes adaptive multi-expert distillation to inherits fusion knowledge from state-of-the-art image fusion models, establishing robust initialization, and then employs a decoupled representation learning scheme to achieve task-relevant information fusion. Moreover, to overcome appearance variations between the initial template and search frames, we presents a nearest-neighbor dynamic template updating scheme, which selects the most reliable frame closest to the current search frame as the dynamic template. Extensive experiments demonstrate that TPF significantly outperforms existing most of advanced trackers on four public RGBT tracking datasets. The code will be released upon acceptance.

Figures (8)

• Figure 1: Analysis of key contradictions in RGBT tracking. (a) Modality gap vs. performance: Comparison of the modality gap at the final backbone layer across trackers with different performance levels. (b) Modality gap vs. interaction location: Comparison of five RGBT variants ostrack with identical interaction modules TBSI but different interaction locations (0/3/6/9/12), analyzing the modality gap at the interaction and final layers, along with tracking performance. (c) Visual comparison of fusion methods: Fusion features at the final layer for pixel-level fusion, feature-level fusion, and the proposed method. MMD values are computed on the entire LasHeR li2021lasher test set.
• Figure 2: Overall framework of Task-driven Pixel-level Fusion (TPF) network for RGBT tracking.
• Figure 3: Visualization of fused images and fused features between different components.
• Figure 4: Visualization of fused images, fused features, and decoupled features with and without DRF. Green boxes indicate task-irrelevant features, while red boxes highlight task-relevant features.
• Figure 5: Comparison of fusion results between the pixel-level fusion method and the other three image fusion methods on the LasHeR dataset.