Uni-MDTrack: Learning Decoupled Memory and Dynamic States for Parameter-Efficient Visual Tracking in All Modality

Abstract

With the advent of Transformer-based one-stream trackers that possess strong capability in inter-frame relation modeling, recent research has increasingly focused on how to introduce spatio-temporal context. However, most existing methods rely on a limited number of historical frames, which not only leads to insufficient utilization of the context, but also inevitably increases the length of input and incurs prohibitive computational overhead. Methods that query an external memory bank, on the other hand, suffer from inadequate fusion between the retrieved spatio-temporal features and the backbone. Moreover, using discrete historical frames as context overlooks the rich dynamics of the target. To address the issues, we propose Uni-MDTrack, which consists of two core components: Memory-Aware Compression Prompt (MCP) module and Dynamic State Fusion (DSF) module. MCP effectively compresses rich memory features into memory-aware prompt tokens, which deeply interact with the input throughout the entire backbone, significantly enhancing the performance while maintaining a stable computational load. DSF complements the discrete memory by capturing the continuous dynamic, progressively introducing the updated dynamic state features from shallow to deep layers, while also preserving high efficiency. Uni-MDTrack also supports unified tracking across RGB, RGB-D/T/E, and RGB-Language modalities. Experiments show that in Uni-MDTrack, training only the MCP, DSF, and prediction head, keeping the proportion of trainable parameters around 30%, yields substantial performance gains, achieves state-of-the-art results on 10 datasets spanning five modalities. Furthermore, both MCP and DSF exhibit excellent generality, functioning as plug-and-play components that can boost the performance of various baseline trackers, while significantly outperforming existing parameter-efficient training approaches.

Figures (9)

• Figure 1: Comparison of existing methods for introducing spatio-temporal context features to one-stream trackers, including the parameter-efficient fine-tuning paradigms. (a) Introducing a memory bank, where the memory features are fused before the prediction head. (b) Introducing auxiliary templates and temporally propagated tokens. (c) Introducing memory-aware compressed tokens as prompts at the model input, and incorporating dynamic state features of the target into the multi-stage layers of the backbone.
• Figure 2: The overall architecture of Uni-MDTrack. Uni-MDTrack can uniformly process data from various modalities and consists of unified modality embedding layer, feature extraction network and prediction head.
• Figure 3: Detail structure of our proposed Memory-Aware Compression Prompt module (MCP) and Dynamic State Fusion module (DSF).
• Figure 4: Comparisons of our proposed Uni-MDTrack with other excellent trackers in the success curve on LaSOT test split, which includes eleven challenging scenarios such as Low Resolution, Motion Blur, Scale Variation, etc. We also provide the comparisons of the success and precision curves across the entire LaSOT test split. Zoom in for better view.
• Figure 5: This figure presents a visual comparison among our proposed Uni-MDTrack, MambaLCT$_{256}$li2025mambalct and SUTrack-B sutrack in the challenges of target among similar objects, undergoes sudden movements, partial occlusion and scale variation. It demonstrates that our method achieves more effective and accurate tracking in the aforementioned challenging scenarios. Zoom in for better view.