CRT-Fusion: Camera, Radar, Temporal Fusion Using Motion Information for 3D Object Detection

TL;DR

This paper tackles robust 3D object detection for autonomous systems by addressing motion awareness in radar-camera BEV fusion. It introduces CRT-Fusion, a framework built from three modules: Multi-View Fusion (MVF) to fuse radar and image features in perspective and BEV spaces; Motion Feature Estimator (MFE) to predict per-pixel velocity and BEV occupancy; and Motion Guided Temporal Fusion (MGTF) to align and fuse BEV features across time using a memory bank. The method achieves state-of-the-art results on the nuScenes radar-camera benchmark, notably surpassing previous best by +1.7% in NDS and +1.4% in mAP, and demonstrates robustness across diverse weather and lighting conditions. While the temporal fusion improves detection accuracy, the approach incurs higher computation with more past frames, motivating future work on recurrent fusion architectures to balance performance and efficiency for real-time deployment.

Abstract

Accurate and robust 3D object detection is a critical component in autonomous vehicles and robotics. While recent radar-camera fusion methods have made significant progress by fusing information in the bird's-eye view (BEV) representation, they often struggle to effectively capture the motion of dynamic objects, leading to limited performance in real-world scenarios. In this paper, we introduce CRT-Fusion, a novel framework that integrates temporal information into radar-camera fusion to address this challenge. Our approach comprises three key modules: Multi-View Fusion (MVF), Motion Feature Estimator (MFE), and Motion Guided Temporal Fusion (MGTF). The MVF module fuses radar and image features within both the camera view and bird's-eye view, thereby generating a more precise unified BEV representation. The MFE module conducts two simultaneous tasks: estimation of pixel-wise velocity information and BEV segmentation. Based on the velocity and the occupancy score map obtained from the MFE module, the MGTF module aligns and fuses feature maps across multiple timestamps in a recurrent manner. By considering the motion of dynamic objects, CRT-Fusion can produce robust BEV feature maps, thereby improving detection accuracy and robustness. Extensive evaluations on the challenging nuScenes dataset demonstrate that CRT-Fusion achieves state-of-the-art performance for radar-camera-based 3D object detection. Our approach outperforms the previous best method in terms of NDS by +1.7%, while also surpassing the leading approach in mAP by +1.4%. These significant improvements in both metrics showcase the effectiveness of our proposed fusion strategy in enhancing the reliability and accuracy of 3D object detection.

Figures (6)

• Figure 1: Comparison of temporal fusion methods: (a) Previous methods concatenate BEV feature maps without considering object motion. (b) CRT-Fusion estimates and compensates for object motion before concatenation. (c) Performance gain of CRT-Fusion over the direct concatenation method, showing CRT-Fusion's superior accuracy across different object velocity ranges.
• Figure 2: Overall architecture of CRT-Fusion: Features are extracted from radar and camera data using backbone networks at each timestamp. The MVF module combines these features to generate fused BEV feature maps. The MFE module predicts the location and velocity of dynamic objects from these maps. The MGTF module then uses the predicted motion information to create the final feature map for the current timestamp, which is fed into the 3D detection head.
• Figure 3: Core components of CRT-Fusion: (a) RCA module enhances image features with radar features for accurate depth prediction. (b) MGTF module compensates for object motion across multiple frames, producing the final BEV feature map for 3D object detection.
• Figure 4: Qualitative results comparing CRT-Fusion and CRN: Green boxes indicate prediction boxes and red boxes represent ground truth (GT) boxes.
• Figure 5: Comparison of velocity prediction using the MFE module in BEVDepth and CRT-Fusion. Red boxes are the Ground Truth (GT) boxes, red arrows show GT velocity, and white arrows represent predicted velocity. Yellow highlights indicate areas where CRT-Fusion predicts velocity more accurately, while orange highlights show static objects correctly identified by CRT-Fusion but misclassified by BEVDepth.