L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection

TL;DR

The paper tackles the challenge of reliable 3D object detection when LiDAR data deteriorates in adverse weather by fusing LiDAR with weather-robust 4D radar. It introduces L4DR, a two-stage fusion framework featuring Foreground-Aware Denoising (FAD) and a Multi-Modal Encoder (MME) for early cross-modal fusion, followed by an Inter-Modal and Intra-Modal ({IM}$^2$) backbone with Multi-Scale Gated Fusion (MSGF) to adaptively fuse features under varying degradation. Key contributions include bridging the data-quality gap at the encoder via MME, mitigating radar noise with FAD, and robust feature fusion with the IM$^2$ backbone and MSGF; jointly, these yield state-of-the-art performance on VoD and K-Radar across fog and adverse weather scenarios, with up to 20% gains in $AP_{3D}$ over LiDAR-only baselines. The results demonstrate practical impact for weather-robust autonomous perception, enabling more reliable navigation in real-world conditions, while acknowledging computational overhead and ~10 FPS inference as areas for optimization. Overall, L4DR establishes a principled, two-stage fusion approach that leverages the complementary strengths of LiDAR and 4D radar to maintain high detection performance when weather challenges degrade one modality.

Abstract

LiDAR-based vision systems are integral for 3D object detection, which is crucial for autonomous navigation. However, they suffer from performance degradation in adverse weather conditions due to the quality deterioration of LiDAR point clouds. Fusing LiDAR with the weather-robust 4D radar sensor is expected to solve this problem. However, the fusion of LiDAR and 4D radar is challenging because they differ significantly in terms of data quality and the degree of degradation in adverse weather. To address these issues, we introduce L4DR, a weather-robust 3D object detection method that effectively achieves LiDAR and 4D Radar fusion. Our L4DR includes Multi-Modal Encoding (MME) and Foreground-Aware Denoising (FAD) technique to reconcile sensor gaps, which is the first exploration of the complementarity of early fusion between LiDAR and 4D radar. Additionally, we design an Inter-Modal and Intra-Modal ({IM}2 ) parallel feature extraction backbone coupled with a Multi-Scale Gated Fusion (MSGF) module to counteract the varying degrees of sensor degradation under adverse weather conditions. Experimental evaluation on a VoD dataset with simulated fog proves that L4DR is more adaptable to changing weather conditions. It delivers a significant performance increase under different fog levels, improving the 3D mAP by up to 20.0% over the traditional LiDAR-only approach. Moreover, the results on the K-Radar dataset validate the consistent performance improvement of L4DR in real-world adverse weather conditions.

Figures (9)

• Figure 1: (a) The radar chart illustrates the complementary nature of LiDAR and 4D radar sensors. (b) AP gaps between LiDAR and 4D radar in real different weather (the extent to which LiDAR is superior to 4D radar).
• Figure 2: (a) Significant quality disparity between the LiDAR and the 4D radar. (b) Severe degradation of LiDAR data quality in adverse weather. (c) Comparison of previous LiDAR-4DRadar fusion and our fusion, highlighting our innovative framework designs to address challenges (a) and (b).
• Figure 3: Performance comparison of our L4DR and LiDAR-only in (a) various simulated fog levels (FL denotes fog level) and (b) real-world adverse weather.
• Figure 4: L4DR framework. (a) Foreground-Aware Denoising (FAD) performs denoising by segmenting foreground semantics per 4D radar point. Next, (b) Multi-Modal Encoder (MME) fuses bi-directional data for both LiDAR and 4D radar modalities at the Encoder stage to obtain higher quality BEV features. Finally, (c) Inter-Modal and Intra-Modal ({IM}$^2$) backbone coupled with Multi-Scale Gated Fusion (MSGF) uses a gating strategy to filter features to avoid redundant information while extracting inter-modal and intra-modal features in parallel.
• Figure 5: Bidirectional Data Fusion in MME. LiDAR-specific point features (blue) and radar-specific point features (red) located in the same pillar are averaged (Avg.) for feature propagation. And the cross-modal offsets (CMO) are computed to enrich the geometric features.