Collaborative Multi-Object Tracking with Conformal Uncertainty Propagation

TL;DR

This work tackles the challenge of exploiting uncertainty from collaborative object detection (COD) to improve multi-object tracking (MOT) in autonomous systems. It introduces MOT-CUP, a framework that quantifies COD uncertainty via direct modeling and conformal prediction (CP) and propagates this uncertainty into the MOT pipeline through a Standard Deviation-based Kalman Filter (SDKF) and a Negative Log Likelihood-based association (NLLAI). On the V2X-Sim dataset, MOT-CUP yields gains in tracking accuracy (e.g., up to 0.85 in HOTA, 1.13 in MOTA, 1.03 in MOTP) and substantially reduces uncertainty (up to 41× NLL improvement and 37% CRPS reduction), with larger improvements under high occlusion. The method is compatible with various COD detectors and MOT baselines, demonstrating the practical value of calibrated uncertainty in perception pipelines and setting a path for broader adoption and extension to other benchmarks.

Abstract

Object detection and multiple object tracking (MOT) are essential components of self-driving systems. Accurate detection and uncertainty quantification are both critical for onboard modules, such as perception, prediction, and planning, to improve the safety and robustness of autonomous vehicles. Collaborative object detection (COD) has been proposed to improve detection accuracy and reduce uncertainty by leveraging the viewpoints of multiple agents. However, little attention has been paid to how to leverage the uncertainty quantification from COD to enhance MOT performance. In this paper, as the first attempt to address this challenge, we design an uncertainty propagation framework called MOT-CUP. Our framework first quantifies the uncertainty of COD through direct modeling and conformal prediction, and propagates this uncertainty information into the motion prediction and association steps. MOT-CUP is designed to work with different collaborative object detectors and baseline MOT algorithms. We evaluate MOT-CUP on V2X-Sim, a comprehensive collaborative perception dataset, and demonstrate a 2% improvement in accuracy and a 2.67X reduction in uncertainty compared to the baselines, e.g. SORT and ByteTrack. In scenarios characterized by high occlusion levels, our MOT-CUP demonstrates a noteworthy $4.01\%$ improvement in accuracy. MOT-CUP demonstrates the importance of uncertainty quantification in both COD and MOT, and provides the first attempt to improve the accuracy and reduce the uncertainty in MOT based on COD through uncertainty propagation. Our code is public on https://coperception.github.io/MOT-CUP/.

Figures (3)

• Figure 1: Difference in data association for MOT with and without considering uncertainty. Ground truth bounding boxes are in green, detected bounding boxes in orange, and tracklets' bounding boxes in red, labeled with object IDs. Shadow ellipses indicate uncertainty of the detected bounding box. SORT bewley2016simple, which doesn't consider uncertainty, is on the left side of the figure, while our MOT-CUP framework, which incorporates uncertainty, is on the right side. At time (t-1), both MOT algorithms output tracklet ID 186. However, at time t, SORT fails to associate the low-quality detection with tracklet 186 due to a large IoU distance. Thus, SORT removes the tracklet. In contrast, our MOT-CUP framework quantifies the uncertainty of COD with a larger shadow ellipse to represent the uncertainty of the bounding box for tracklet 186, and successfully associates the low-quality detection by considering the uncertainty of COD.
• Figure 2: Overview of our MOT-CUP framework. The red color highlights the novelties and important techniques in our MOT-CUP framework. In the collaborative object detection (COD) stage, we rigorously calculate uncertainty quantification (UQ) of each object detection via direct modeling (DM) and conformal prediction (CP). In the motion prediction stage of MOT, we adopt a Standard Deviation-based Kalman Filter (SDKF) to enhance the Kalman Filter process, that leverages the UQ results and predicts the locations of the objects in the next time step with higher precision. In the association step, we first apply the baseline association method and then associate the unmatched detections and tracklets with the Negative Log Likelihood similarity metric, called NLLAI.
• Figure 3: Visualization of results of the detection, original SORT, and our MOT-CUP framework over consecutive three frames. The collaborative object detector here is Upper-bound. Green boxes are ground truth bounding boxes, orange boxes are detected bounding boxes, and red boxes are tracklets' bounding boxes as the output of MOT. The numbers next to the red boxes indicate object IDs. We observe that our MOT-CUP outperforms the original SORT algorithm in tracking object 332, as indicated by the red arrow. Furthermore, MOT-CUP improves the accuracy of location, compared with the object detector, such as object 332 in frame 60. Overall, our results demonstrate the importance of considering uncertainty in MOT.

Theorems & Definitions (3)

• Lemma III.1: Conformal Coverage Guarantee angelopoulos2021gentle
• Proposition 1
• proof