DM$^3$T: Harmonizing Modalities via Diffusion for Multi-Object Tracking

TL;DR

DM^3T reframes RGB-thermal multi-object tracking as iterative, diffusion-inspired cross-modal feature alignment. It introduces Cross-Modal Diffusion Fusion to harmonize RGB and thermal features, followed by a Diffusion Refiner and a Hierarchical Tracker to achieve robust online tracking without post-processing. On VT-MOT, it delivers state-of-the-art HOTA (41.7) and IDF1 (48.0), with real-time performance around 15 FPS, at the cost of some false positives and identity switches. The approach highlights the power of diffusion-based cross-modal fusion for robust multimodal perception while acknowledging trade-offs between comprehensive association and precision metrics, guiding future refinement toward uncertainty-aware diffusion.

Abstract

Multi-object tracking (MOT) is a fundamental task in computer vision with critical applications in autonomous driving and robotics. Multimodal MOT that integrates visible light and thermal infrared information is particularly essential for robust autonomous driving systems. However, effectively fusing these heterogeneous modalities is challenging. Simple strategies like concatenation or addition often fail to bridge the significant non-linear distribution gap between their feature representations, which can lead to modality conflicts and degrade tracking accuracy. Drawing inspiration from the connection between multimodal MOT and the iterative refinement in diffusion models, this paper proposes DM$^3$T, a novel framework that reformulates multimodal fusion as an iterative feature alignment process to generate accurate and temporally coherent object trajectories. Our approach performs iterative cross-modal harmonization through a proposed Cross-Modal Diffusion Fusion (C-MDF) module. In this process, features from both modalities provide mutual guidance, iteratively projecting them onto a shared, consistent feature manifold. This enables the learning of complementary information and achieves deeper fusion compared to conventional methods. Additionally, we introduce a plug-and-play Diffusion Refiner (DR) to enhance and refine the unified feature representation. To further improve tracking robustness, we design a Hierarchical Tracker that adaptively handles confidence estimation. DM$^3$T unifies object detection, state estimation, and data association into a comprehensive online tracking framework without complex post-processing. Extensive experiments on the VT-MOT benchmark demonstrate that our method achieves 41.7 HOTA, representing a 1.54% relative improvement over existing state-of-the-art methods. The code and models are available at https://vranlee.github.io/DM-3-T/.

Figures (6)

• Figure 1: Performance comparison on the VT-MOT test set. Our DM$^3$T achieves state-of-the-art performance with HOTA of 41.7 and IDF1 of 48.0. The subfigure presents results across additional evaluation metrics. Detailed quantitative results are provided in Table \ref{['tab:tracking_performance']}.
• Figure 2: Overall architecture of DM$^3$T. The framework consists of a Cross-Modal Diffusion Fusion module that performs iterative cross-modal harmonization between RGB and thermal features, a Diffusion Refiner for further feature enhancement, and a Hierarchical Tracker that manages object associations through confidence-guided multi-stage association and adaptive motion prediction.
• Figure 3: The pipeline of Cross-Modal Diffusion Fusion (C-MDF). The module consists of iterative cross-modal harmonization blocks that perform controlled perturbation and refinement to progressively align RGB and thermal features. Each refinement network processes concatenated features from both modalities, enabling deep cross-modal interaction.
• Figure 4: Qualitative comparison of feature maps, with corresponding quantitative analysis in Table \ref{['tab:qual_metrics']}. Additional tracking visualizations are provided in the Supplementary Material.
• Figure 5: Tracking performance on LashHer-020 sequence. Our method maintains robust multi-object tracking despite significant occlusion. Different object identities are distinguished by bounding box colors and ID numbers, best viewed in color and zoomed. Same as below.