RockTrack: A 3D Robust Multi-Camera-Ken Multi-Object Tracking Framework

TL;DR

RockTrack addresses the challenges of 3D MOT with multi-camera detectors by adopting a Tracking-By-Detection framework that remains detector-agnostic. It introduces a confidence-guided pre-processing pipeline and a novel Multi-Camera Appearance Similarity (MCAS) metric, combined with a noise-adaptive motion model and a lifecycle module, to robustly fuse motion and appearance across views. The method achieves state-of-the-art vision-only AMOTA on nuScenes (59.1%) without detector-specific training and runs efficiently on CPU, demonstrating strong robustness to sensor dropout and practical applicability. Overall, RockTrack provides a scalable, detector-agnostic baseline for multi-camera 3D MOT that leverages multi-view appearance and geometry without end-to-end retraining.

Abstract

3D Multi-Object Tracking (MOT) obtains significant performance improvements with the rapid advancements in 3D object detection, particularly in cost-effective multi-camera setups. However, the prevalent end-to-end training approach for multi-camera trackers results in detector-specific models, limiting their versatility. Moreover, current generic trackers overlook the unique features of multi-camera detectors, i.e., the unreliability of motion observations and the feasibility of visual information. To address these challenges, we propose RockTrack, a 3D MOT method for multi-camera detectors. Following the Tracking-By-Detection framework, RockTrack is compatible with various off-the-shelf detectors. RockTrack incorporates a confidence-guided preprocessing module to extract reliable motion and image observations from distinct representation spaces from a single detector. These observations are then fused in an association module that leverages geometric and appearance cues to minimize mismatches. The resulting matches are propagated through a staged estimation process, forming the basis for heuristic noise modeling. Additionally, we introduce a novel appearance similarity metric for explicitly characterizing object affinities in multi-camera settings. RockTrack achieves state-of-the-art performance on the nuScenes vision-only tracking leaderboard with 59.1% AMOTA while demonstrating impressive computational efficiency.

Figures (3)

• Figure 1: HeightTrans li2024dualbev detections highlight two key features of multi-camera detectors. (I) The inherent ill-posed nature results in unreliable spatial estimations, i.e., 0.4 score $FP_{3D}$. (II) The imprecise yet rich feature descriptions across various representation spaces allow 3D low-score detections, i.e., 0.12 score $FP_{3D}$, to be valid visual residuals for certain trajectories.
• Figure 2: (a): The pipeline of RockTrack. (I): Previous tracklets $T_{t-1}$ are predicted to $T_{t, t-1}$. (II): 3D raw detections $D^{r}_{t}$ undergo a geometry filter to reduce FP, resulting in $D_{t}$. A visual filter is then introduced to recall valid low-score detections in $D_{t}$, obtaining $D^{low}_{t}$. (III): High-score detections $D^{high}_{t}$ in $D_{t}$ are first matched with $T_{t, t-1}$ based on motion. $D^{low}_{t}$, unmatched detections in $D^{high}_{t}$ and tracklets, are then associated through appearance. (IV): Final matched detections and heuristic observation-specific noises are utilized to update the corresponding tracklets. (V): The count-based lifecycle module is employed to init, penalize, and discard tracklets. All alive tracklets merge to obtain $T_{t}$ and forward to the subsequent frame.(b): The calculation process of our proposed multi-camera appearance similarity metric (MCAS). ($K=6$)
• Figure 3: The comparison of the accuracy under distinct hyperparameter. All categories are applied to the same parameter.