Radar-Camera Fused Multi-Object Tracking: Online Calibration and Common Feature

TL;DR

The paper tackles robust multi-object tracking in autonomous driving by integrating radar and camera data with online calibration based on common features, treating radar as the primary source for range/position. It introduces a Common Feature Discriminator to learn cross-modal representations for online, targetless calibration via planar homography, enabling direct mapping of camera detections to the radar plane. A two-stage sensor fusion approach—first feature-based matching then distance-based matching—along with a category-consistency check yields more accurate cross-sensor associations and inter-frame tracking stability. The method demonstrates strong performance in both controlled and real-world experiments, including resilience to sensor failure and challenging weather, with code availability for reproducibility.

Abstract

This paper presents a Multi-Object Tracking (MOT) framework that fuses radar and camera data to enhance tracking efficiency while minimizing manual interventions. Contrary to many studies that underutilize radar and assign it a supplementary role--despite its capability to provide accurate range/depth information of targets in a world 3D coordinate system--our approach positions radar in a crucial role. Meanwhile, this paper utilizes common features to enable online calibration to autonomously associate detections from radar and camera. The main contributions of this work include: (1) the development of a radar-camera fusion MOT framework that exploits online radar-camera calibration to simplify the integration of detection results from these two sensors, (2) the utilization of common features between radar and camera data to accurately derive real-world positions of detected objects, and (3) the adoption of feature matching and category-consistency checking to surpass the limitations of mere position matching in enhancing sensor association accuracy. To the best of our knowledge, we are the first to investigate the integration of radar-camera common features and their use in online calibration for achieving MOT. The efficacy of our framework is demonstrated by its ability to streamline the radar-camera mapping process and improve tracking precision, as evidenced by real-world experiments conducted in both controlled environments and actual traffic scenarios. Code is available at https://github.com/radar-lab/Radar_Camera_MOT

Figures (15)

• Figure 1: Visualization of Errors Caused by Pitch Angle Change.
• Figure 2: Target-Based Calibration Scenarios from domhof2019extrinsic and olutomilayo2021extrinsic typically require specific targets and environments, as well as manual intervention.
• Figure 3: Framework of the Proposed Radar-Camera Fusion MOT Method.
• Figure 4: Common Feature Discriminator Model Architecture. CSPResNet: Cross-Stage Partial ResNet, CBL: Convolution3D + Batch Normalization + LeakyReLU, SPP: Spatial Pyramid Pooling, PANet: Path Aggregation Network, are from YOLO bochkovskiy2020yolov4.
• Figure 5: Example Frames Demonstrating Multi-Object Tracking Results of the Proposed Method. Camera Branch Results are Obtained by Projecting Camera Detections onto the Radar Plane.