SDCM: Simulated Densifying and Compensatory Modeling Fusion for Radar-Vision 3-D Object Detection in Internet of Vehicles

TL;DR

SDCM tackles sparse 4-D radar data and vision degradation in IoV by integrating SimDen for radar densification, RCM for vision degradation compensation, and MMIF for inter-modal interaction. The approach leverages KDE-based key-point densification, multi-scale radar-vision attention, and Mamba-driven fusion to produce robust 3-D object detection. Across VoD, TJ4DRadSet, and Astyx HiRes 2019, SDCM achieves superior accuracy with lower parameter counts and faster inference, demonstrating strong generalization and practical impact for all-weather, real-time autonomous driving. The work provides comprehensive ablations and visualizations to validate the effectiveness of each module and their synergy.

Abstract

3-D object detection based on 4-D radar-vision is an important part in Internet of Vehicles (IoV). However, there are two challenges which need to be faced. First, the 4-D radar point clouds are sparse, leading to poor 3-D representation. Second, vision datas exhibit representation degradation under low-light, long distance detection and dense occlusion scenes, which provides unreliable texture information during fusion stage. To address these issues, a framework named SDCM is proposed, which contains Simulated Densifying and Compensatory Modeling Fusion for radar-vision 3-D object detection in IoV. Firstly, considering point generation based on Gaussian simulation of key points obtained from 3-D Kernel Density Estimation (3-D KDE), and outline generation based on curvature simulation, Simulated Densifying (SimDen) module is designed to generate dense radar point clouds. Secondly, considering that radar data could provide more real time information than vision data, due to the all-weather property of 4-D radar. Radar Compensatory Mapping (RCM) module is designed to reduce the affects of vision datas' representation degradation. Thirdly, considering that feature tensor difference values contain the effective information of every modality, which could be extracted and modeled for heterogeneity reduction and modalities interaction, Mamba Modeling Interactive Fusion (MMIF) module is designed for reducing heterogeneous and achieving interactive Fusion. Experiment results on the VoD, TJ4DRadSet and Astyx HiRes 2019 dataset show that SDCM achieves best performance with lower parameter quantity and faster inference speed. Our code will be available.

Figures (8)

• Figure 1: The connection between IoV and 3-D object detection. The communication of correct detection results provides transportation safty information for the vehicles, which is beneficial for making correct driving decisions.
• Figure 2: The challeges of 3-D object detection based on 4-D radar-vision fusion. In the first row, (a), (b) and (c) shows the sparsity of 4-D radar point clouds, the instances have few points generated by 4-D radar sensors. The second row shows the missed detection of state-of-the-art methods, which are nder the representation degradation of vision datas. (d) is low-light, (e) is long distance detection, and (f) is dense occlusion. The red bounding boxes are GroundTruth, while other color bounding boxes are the detection results. These images come from VoD, TJ4DRadSet and Astyx HiRes 2019 dataset.
• Figure 3: The overall pipeline of SDCM. (a) contains SimDen module, as well as the backbones of radar point clouds and RGB images. (b) is VMamba architecture which is adopted as image backbone. (c) is RCM module, which adopts radar features to compensate the represenstation degradation of vision datas. (d) is MMIF module, which reduce heterogeneous features and achieve feature interaction fusion.
• Figure 4: The overall architecture of SimDen module. The dense radar point clouds are simulated through two steps, one is 3-D KDE and Gaussian simulation, the other is curvature estiamtion and simulation.
• Figure 5: The overall architecture of fusion stage in SDCM. (a) is the program of the fusion stage. (b) is the RCM module, which achieves the compensation of representation degradation. (c) is the MMIF module, which reduces heterogeneous features and achieves feature interaction fusion.