SwiTrack: Tri-State Switch for Cross-Modal Object Tracking

TL;DR

Cross-modal object tracking with dynamic modality switches (RGB/NIR) faces feature-distribution mismatches and drift when inputs degrade. SwiTrack introduces a tri-state switch to jointly handle RGB, NIR, and invalid inputs, a NIR gated adapter to align NIR features with the shared backbone, and a Consistent Trajectory Prediction module to maintain tracking during invalid states; it also employs dynamic template reconstruction and a modality-consistent loss to reinforce cross-modal alignment. On CMOTB, SwiTrack attains state-of-the-art Precision Rate and Success Rate with significant gains over competitive baselines, while maintaining real-time performance at 65 FPS. These advances yield robust cross-modal tracking suitable for real-world surveillance and robotics where modality reliability fluctuates and rapid response is required.

Abstract

Cross-modal object tracking (CMOT) is an emerging task that maintains target consistency while the video stream switches between different modalities, with only one modality available in each frame, mostly focusing on RGB-Near Infrared (RGB-NIR) tracking. Existing methods typically connect parallel RGB and NIR branches to a shared backbone, which limits the comprehensive extraction of distinctive modality-specific features and fails to address the issue of object drift, especially in the presence of unreliable inputs. In this paper, we propose SwiTrack, a novel state-switching framework that redefines CMOT through the deployment of three specialized streams. Specifically, RGB frames are processed by the visual encoder, while NIR frames undergo refinement via a NIR gated adapter coupled with the visual encoder to progressively calibrate shared latent space features, thereby yielding more robust cross-modal representations. For invalid modalities, a consistency trajectory prediction module leverages spatio-temporal cues to estimate target movement, ensuring robust tracking and mitigating drift. Additionally, we incorporate dynamic template reconstruction to iteratively update template features and employ a similarity alignment loss to reinforce feature consistency. Experimental results on the latest benchmarks demonstrate that our tracker achieves state-of-the-art performance, boosting precision rate and success rate gains by 7.2\% and 4.3\%, respectively, while maintaining real-time tracking at 65 frames per second. Code and models are available at https://github.com/xuboyue1999/SwiTrack.git.

Figures (11)

• Figure 1: Framework comparisons between existing and proposed cross-modal trackers. (a) Existing tracker: Utilizes symmetric dual branches following the visual encoder, which limits the extraction of modality-specific features and lacks robustness against modality invalidity due to over-exposure. (b) SwiTrack: Designs a tri-state switch for dynamic state assessment, a NIR gated adapter for feature modulation, and consistent trajectory prediction for robust motion cue modeling during switching.
• Figure 2: The framework of the proposed SwiTrack. Templates, the search region, dynamic template features, and trajectory are tokenized and processed by an $l$-layer ViT block. It identifies the search region’s state, RGB, NIR, or invalid, and activates the NIR gated adapter for modality adaptation. The template reconstruction module updates dynamic templates using current visual features, while the prediction head generates tracking results. When the state is invalid, the consistent trajectory prediction module estimates the trajectory without relying on visual features. State switch includes modality change detection based on features and invalid detection based on the image.
• Figure 3: Detailed design of the NIR gated adapter, which aims to align the NIR modality with the original features after state switching.
• Figure 4: Detailed design of the consistency trajectory prediction, which aims to predict the tracking trajectory when over-exposure occurs.
• Figure 5: The performance of different trackers on the CMOTB testing set.