Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking

TL;DR

Diff-MM introduces a unified multi-modal object tracker that leverages the pre-trained Stable Diffusion UNet as a tracking feature extractor. Through a parallel feature extraction pipeline (PFE) and multi-modal sub-module tuning (MST), it handles RGB-N/D/T/E inputs with a single parameter set, avoiding architectural changes to the UNet. Two-stage training jointly fine-tunes RGB and RGB-N features and then incorporates auxiliary modalities via a cloned encoder/middle block, while freezing the UNet and head. The method achieves strong, consistent gains across RGB-N/D/T/E benchmarks (e.g., 8.3% AUC on TNL2K and 4.9% higher F-score on DepthTrack), demonstrating the practical value of diffusion priors for cross-modal tracking with limited multi-modal data.

Abstract

Multi-modal object tracking integrates auxiliary modalities such as depth, thermal infrared, event flow, and language to provide additional information beyond RGB images, showing great potential in improving tracking stabilization in complex scenarios. Existing methods typically start from an RGB-based tracker and learn to understand auxiliary modalities only from training data. Constrained by the limited multi-modal training data, the performance of these methods is unsatisfactory. To alleviate this limitation, this work proposes a unified multi-modal tracker Diff-MM by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model. Diff-MM leverages the UNet of pre-trained Stable Diffusion as a tracking feature extractor through the proposed parallel feature extraction pipeline, which enables pairwise image inputs for object tracking. We further introduce a multi-modal sub-module tuning method that learns to gain complementary information between different modalities. By harnessing the extensive prior knowledge in the generation model, we achieve a unified tracker with uniform parameters for RGB-N/D/T/E tracking. Experimental results demonstrate the promising performance of our method compared with recently proposed trackers, e.g., its AUC outperforms OneTracker by 8.3% on TNL2K.

Figures (3)

• Figure 1: (a) Recent multi-modal trackers typically start from a foundation RGB tracker trained on large-scale RGB tracking data. They learn to understand auxiliary modality from small-scale multi-modal tracking data, using prompt-tuning or LoRA to avoid forgetting. (b) Our method acquires a unified multi-modal tracker by leveraging a text-to-image generation model. With the help of prior knowledge in the generation model, we achieve a unified tracker for RGB-N/D/T/E tracking.
• Figure 2: Illustrations of the overall framework and the parallel feature extraction pipeline of Diff-MM. (a) We achieve a unified model for RGB-N/D/T/E tracking by exploiting generative priors from the pre-trained SD. For RGB-D/T/E tracking, we first encoder the RGB/Depth/Thermal/Event images into the latent space with the VAE encoder and noise adding. Latent features of RGB images are sent into the UNet. We introduce a multi-modal sub-module copied from the encoder and middle block of the UNet to model auxiliary modalities. Lateral-connection is used to inject auxiliary information into the UNet. The output features are fed into the tracking head for bounding box prediction. The language description of the object is used as the text condition to perform RGB-N tracking. (b) The parallel feature extraction pipeline enables pairwise image inputs by extracting template features and search features in parallel and modeling their relationship at the self attention layer in each Basic Block of the UNet.
• Figure 3: Visualization of tracking results.