ZFusion: An Effective Fuser of Camera and 4D Radar for 3D Object Perception in Autonomous Driving

TL;DR

ZFusion tackles robust 3D object perception for autonomous driving by fusing sparse 4D radar data with dense vision features. It introduces FP-DDCA, a pyramid of deformable cross-attention blocks, together with a Depth-Context-Split view transformation to align modalities in BEV space. The DDCA transformer block balances information across radar and camera while mitigating modality-order bias, and the FP-DDCA fuser enables multi-scale fusion for improved accuracy. On the VoD dataset, ZFusion achieves state-of-the-art RoI mAP and competitive entire-scene performance, demonstrating that a cost-effective radar–camera system can approach LiDAR-based perception, with robustness to adverse weather and potential for broader deployment.

Abstract

Reliable 3D object perception is essential in autonomous driving. Owing to its sensing capabilities in all weather conditions, 4D radar has recently received much attention. However, compared to LiDAR, 4D radar provides much sparser point cloud. In this paper, we propose a 3D object detection method, termed ZFusion, which fuses 4D radar and vision modality. As the core of ZFusion, our proposed FP-DDCA (Feature Pyramid-Double Deformable Cross Attention) fuser complements the (sparse) radar information and (dense) vision information, effectively. Specifically, with a feature-pyramid structure, the FP-DDCA fuser packs Transformer blocks to interactively fuse multi-modal features at different scales, thus enhancing perception accuracy. In addition, we utilize the Depth-Context-Split view transformation module due to the physical properties of 4D radar. Considering that 4D radar has a much lower cost than LiDAR, ZFusion is an attractive alternative to LiDAR-based methods. In typical traffic scenarios like the VoD (View-of-Delft) dataset, experiments show that with reasonable inference speed, ZFusion achieved the state-of-the-art mAP (mean average precision) in the region of interest, while having competitive mAP in the entire area compared to the baseline methods, which demonstrates performance close to LiDAR and greatly outperforms those camera-only methods.

Figures (5)

• Figure 1: ZFusion pipeline based on FP-DDCA fuser.
• Figure 2: Architecture of ZFusion, including the DCA mechanism (left-bottom), DDCA block (right), and FP-DDCA fuser (left-top).
• Figure 3: 4D radar can obtain some 'X-Ray vision' point clouds, shown in BEV space.
• Figure 4: Typical visualizations in VoD test set, ID: 0262 (left) and 4942 (right)
• Figure 5: Feature maps of different modalities and feature maps fused by Convolution and FP-DDCA.