Adaptive Perception for Unified Visual Multi-modal Object Tracking

TL;DR

APTrack tackles the problem of underutilization of auxiliary modalities in unified multi-modal tracking by enforcing equal modeling across modalities and introducing adaptive perception. The method integrates an Adaptive Modality Interaction (AMI) module and a global modal preceptor to enable efficient cross-modal fusion with learnable tokens, without task-specific fine-tuning. Experiments on five datasets (RGBT234, LasHeR, VisEvent, DepthTrack, VOT-RGBD2022) show that APTrack achieves state-of-the-art performance among unified trackers and outperforms some task-specific trackers. This approach improves perception robustness in challenging conditions such as low-light, occlusion, and rapid motion, offering a practical, unified solution for multi-modal tracking.

Abstract

Recently, many multi-modal trackers prioritize RGB as the dominant modality, treating other modalities as auxiliary, and fine-tuning separately various multi-modal tasks. This imbalance in modality dependence limits the ability of methods to dynamically utilize complementary information from each modality in complex scenarios, making it challenging to fully perceive the advantages of multi-modal. As a result, a unified parameter model often underperforms in various multi-modal tracking tasks. To address this issue, we propose APTrack, a novel unified tracker designed for multi-modal adaptive perception. Unlike previous methods, APTrack explores a unified representation through an equal modeling strategy. This strategy allows the model to dynamically adapt to various modalities and tasks without requiring additional fine-tuning between different tasks. Moreover, our tracker integrates an adaptive modality interaction (AMI) module that efficiently bridges cross-modality interactions by generating learnable tokens. Experiments conducted on five diverse multi-modal datasets (RGBT234, LasHeR, VisEvent, DepthTrack, and VOT-RGBD2022) demonstrate that APTrack not only surpasses existing state-of-the-art unified multi-modal trackers but also outperforms trackers designed for specific multi-modal tasks.

Figures (8)

• Figure 1: APTrack is a unified multi-modal tracker that can be applied to various RGB-X tasks (such as RGB-Depth, RGB-TIR, and RGB-Event) with a unified parameters.
• Figure 2: The overall structure of APTrack. APTrack is composed of shared embedding, shared transformer block, AMI and Head. The method of modal processing in this model is completely consistent, and there is no need for extra processing for a certain model, which makes the modal features can be aligned adaptively. In addition, AMI can transfer the advantages of modalities to each other.
• Figure 3: The detailed architecture of Global Modal Perceptor. These perceptors use attention mechanisms to enforce multi-modal global attention.
• Figure 4: Visual result of RGB-T. The three sequences from top to bottom represent the following scenarios: both modalities exhibit low effectiveness, the RGB modality is less effective than the TIR modality, and the TIR modality is less effective than the RGB modality.
• Figure 5: More MPR/MSR comparisons on RGBT234.