MCBLT: Multi-Camera Multi-Object 3D Tracking in Long Videos

TL;DR

MCBLT addresses the challenge of robust MTMC tracking in long videos from static multi-camera setups by performing early multi-view aggregation into a BEV representation for 3D object detection, followed by 2D-3D ReID association and a hierarchical GNN-based tracking framework in 3D space. The approach introduces a global tracking block that enables long-range associations across thousands of frames, removing reliance on windowed heuristics. Key contributions include the first 3D MOT with hierarchical GNNs for MTMC, a 2D-3D detection association pipeline for stable ReID features, and state-of-the-art results on AICity'24 ($$81.22$$ HOTA) and WildTrack ($$95.6$$ IDF1). The work demonstrates strong generalization to diverse scenes and camera setups, enabling robust long-term tracking in practical surveillance and robotics applications.

Abstract

Object perception from multi-view cameras is crucial for intelligent systems, particularly in indoor environments, e.g., warehouses, retail stores, and hospitals. Most traditional multi-target multi-camera (MTMC) detection and tracking methods rely on 2D object detection, single-view multi-object tracking (MOT), and cross-view re-identification (ReID) techniques, without properly handling important 3D information by multi-view image aggregation. In this paper, we propose a 3D object detection and tracking framework, named MCBLT, which first aggregates multi-view images with necessary camera calibration parameters to obtain 3D object detections in bird's-eye view (BEV). Then, we introduce hierarchical graph neural networks (GNNs) to track these 3D detections in BEV for MTMC tracking results. Unlike existing methods, MCBLT has impressive generalizability across different scenes and diverse camera settings, with exceptional capability for long-term association handling. As a result, our proposed MCBLT establishes a new state-of-the-art on the AICity'24 dataset with $81.22$ HOTA, and on the WildTrack dataset with $95.6$ IDF1.

Figures (7)

• Figure 1: Comparison among three types of MTMC tracking methods. (a) conducts 2D detection separately and associates objects among different views by appearance-based ReID; (b) considers geometric constraints as well besides appearance for cross-view association; (c) achieves multi-view association in early stage by feature-level aggregation.
• Figure 2: The overall framework of MCBLT. First, multi-view images at frame $t$ are passed through the image backbone to obtain multi-view image features. A spatial encoder is then introduced to aggregate multi-view image features to BEV features $B_t$, followed by a temporal encoder to aggregate BEV features within a temporal window. A DETR-based decoder is utilized to obtain object detection results, which are in the format of 3D bounding boxes. To get reliable ReID features for the detected objects, a ReID feature extraction module is proposed, including a 2D ReID feature extractor and a 2D-3D detection association algorithm. Finally, SUSHI-3D is designed to achieve multi-object tracking in BEV to obtain the final MTMC tracking results. (SUSHI Block graphics are from cetintas2023unifying.)
• Figure 3: Illustration on 2D detection (in blue) and the corresponding projected 3D detection (in green).
• Figure 4: GNN hierarchy of MCBLT for tracking. We process long sequences in a near-online fashion with stride $s$. But in contrast to cetintas2023unifying, we omit overlaps between windows and heuristics. To associate incoming with past tracks, MCBLT uses a global merging block. The global block requires no additional training.
• Figure 5: Visualization of MTMC detection and tracking results for three different scenes in AICity'24 test set. The tracked objects are shown as colored dots in the BEV floor plans, and object 3D bounding boxes are projected and drawn in each camera view.