UBATrack: Spatio-Temporal State Space Model for General Multi-Modal Tracking

TL;DR

UBATrack tackles the challenge of general multi-modal tracking by integrating cross-modal spatio-temporal cues within a mamba-style state-space framework. The method introduces a Spatio-Temporal Mamba Adapter (STMA) for joint cross-modal and temporal modeling and a Dynamic Multi-modal Feature Mixer (DMFM) for robust fusion, all implemented via adapter-tuning to avoid full fine-tuning. Empirical results across six benchmarks (e.g., LasHeR, RGBT234, RGBT210, DepthTrack, VOT-RGBD22, VisEvent) demonstrate state-of-the-art performance and strong robustness in diverse conditions. The combination of STMA and DMFM provides a scalable, efficient approach to cross-modal tracking with practical impact for real-world multi-modal perception tasks.

Abstract

Multi-modal object tracking has attracted considerable attention by integrating multiple complementary inputs (e.g., thermal, depth, and event data) to achieve outstanding performance. Although current general-purpose multi-modal trackers primarily unify various modal tracking tasks (i.e., RGB-Thermal infrared, RGB-Depth or RGB-Event tracking) through prompt learning, they still overlook the effective capture of spatio-temporal cues. In this work, we introduce a novel multi-modal tracking framework based on a mamba-style state space model, termed UBATrack. Our UBATrack comprises two simple yet effective modules: a Spatio-temporal Mamba Adapter (STMA) and a Dynamic Multi-modal Feature Mixer. The former leverages Mamba's long-sequence modeling capability to jointly model cross-modal dependencies and spatio-temporal visual cues in an adapter-tuning manner. The latter further enhances multi-modal representation capacity across multiple feature dimensions to improve tracking robustness. In this way, UBATrack eliminates the need for costly full-parameter fine-tuning, thereby improving the training efficiency of multi-modal tracking algorithms. Experiments show that UBATrack outperforms state-of-the-art methods on RGB-T, RGB-D, and RGB-E tracking benchmarks, achieving outstanding results on the LasHeR, RGBT234, RGBT210, DepthTrack, VOT-RGBD22, and VisEvent datasets.

Figures (7)

• Figure 1: (a) These trackers OneTrackerSDSTrackun-track primarily focus on modality fusion but often overlook effective spatio-temporal cue capture across modalities. (b) These trackers TATrack learn temporal history information through full fine-tuning, which increases the training burden. (c) UBATrack improves tracking performance by leveraging adapter-tuning to jointly model multi-modal spatio-temporal visual cues.
• Figure 2: Overview architecture of UBATrack. UBATrack freezes the encoder and performs adapter-based fine-tuning only on the proposed Spatio-Temporal Mamba Adapter (STMA) and Dynamic Multi-modal Feature Mixer (DMFM). It takes multimodal video clips as input for feature extraction, where X denotes infrared, depth, or event modalities. The learned multi-modal token features are then divided into two groups and fed into STMA to enable cross-modal interaction and spatio-temporal modeling. Finally, the RGB and X modality features are processed by DMFM to enhance their representational capacity, thereby achieving multi-modal object localization.
• Figure 3: Spatio-Temporal Mamba Adapter (STMA) combines cross-modal interactions and spatio-temporal cues for multi-modal tracking. It uses Mamba blocks for token modeling and MultiFFT for channel mixing. MultiFFT, composed of EMM, FFT and IFFT, to boost multi-modal feature extraction.
• Figure 4: Dynamic Multi-modal Feature Mixer fuses cross-modal features using a Mixer Layer, Global Average Pooling, and Channel-MLP. The Mixer Layer combines token and channel information, while MMixOp enhances feature fusion.
• Figure 5: Precision scores of different attributes on the LasHeR LasHeR and VisEvent VisEvent dataset.