MTMMC: A Large-Scale Real-World Multi-Modal Camera Tracking Benchmark

TL;DR

MTMMC addresses the scarcity of large-scale, real-world MTMC benchmarks by introducing a 16-camera RGB+Thermal dataset collected in campus and factory environments, totaling 3,052,800 frames and 3,669 identities under varied conditions. The paper provides a full annotation pipeline, three evaluation sub-tasks (detection, Re-ID, MOT), and extensive baselines, including real-world pre-training versus synthetic data, and two multimodal learning setups (modality fusion and modality drop). Key findings show that thermal information fused at the feature level improves tracking robustness, real-world MTMC pre-training enhances cross-domain generalization, and combining MTMMC with synthetic data yields strong performance gains. The MTMMC dataset, with its privacy-conscious collection and open test server, offers a practical, scalable resource for advancing MTMC research and real-world deployment of multi-modal tracking systems.

Abstract

Multi-target multi-camera tracking is a crucial task that involves identifying and tracking individuals over time using video streams from multiple cameras. This task has practical applications in various fields, such as visual surveillance, crowd behavior analysis, and anomaly detection. However, due to the difficulty and cost of collecting and labeling data, existing datasets for this task are either synthetically generated or artificially constructed within a controlled camera network setting, which limits their ability to model real-world dynamics and generalize to diverse camera configurations. To address this issue, we present MTMMC, a real-world, large-scale dataset that includes long video sequences captured by 16 multi-modal cameras in two different environments - campus and factory - across various time, weather, and season conditions. This dataset provides a challenging test-bed for studying multi-camera tracking under diverse real-world complexities and includes an additional input modality of spatially aligned and temporally synchronized RGB and thermal cameras, which enhances the accuracy of multi-camera tracking. MTMMC is a super-set of existing datasets, benefiting independent fields such as person detection, re-identification, and multiple object tracking. We provide baselines and new learning setups on this dataset and set the reference scores for future studies. The datasets, models, and test server will be made publicly available.

Figures (21)

• Figure 1: The 3D layout overview. (a) campus and (b) factory. We installed 16 multi-modal cameras in both indoor and outdoor settings, across multiple floors, with overlapping coverage. The cameras were fixed in position and angle to densely cover the building, creating a realistic surveillance camera system.
• Figure 2: MTMC Dataset statistics comparison. We compare MTMMC dataset with the current largest simulated MTAkohl2020mta and real world MMPTrackhan2021mmptrack datasets. Each visual summarizes specific statistics of each dataset : (a) The instance tracks by the number of visited cameras, (b) The joint distribution of instance normalized width and height, (c) The instance track centers plotted over normalized image coordinates.
• Figure 3: Multi-modal Learning Setups and Baselines. (a) presents the concept of modality fusion with both input-level and feature-level fusion techniques integrating thermal data with RGB for enhanced object tracking. (b) outlines the modality drop scenario, where the model trained on combined RGB and thermal data is tested solely on RGB data, using methods like multi-modal reconstruction, knowledge distillation, and multi-modal contrastive learning.
• Figure 4: Single-camera tracking annotation pipeline. We adopt the semi-automatic labeling approach. The workers first label the key frames and then the annotations for the other frames are interpolated based on the model predictions.
• Figure 5: Multi-camera association. The workers are instructed to assign consistent PIDs for the same person across the cameras for each scenario.