DepthFusion: Depth-Aware Hybrid Feature Fusion for LiDAR-Camera 3D Object Detection

TL;DR

This work tackles the neglect of depth information in LiDAR-camera 3D object detection by introducing DepthFusion, a depth-aware hybrid feature fusion framework. It implements two modules, Depth-GFusion for depth-guided global BEV fusion and Depth-LFusion for depth-aware local refinement, using a sine-cosine depth encoding to weight modality contributions. Empirical results on nuScenes, KITTI, and nuScenes-C demonstrate state-of-the-art accuracy and enhanced robustness to corruptions, with strong performance across far-distance objects and realistic sensor degradations. The findings highlight the benefit of explicitly modeling depth when fusing multi-modal features for robust 3D perception in autonomous driving contexts.

Abstract

State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion. However, they neglect the factor of depth while designing the fusion strategy. In this work, we are the first to observe that different modalities play different roles as depth varies via statistical analysis and visualization. Based on this finding, we propose a Depth-Aware Hybrid Feature Fusion (DepthFusion) strategy that guides the weights of point cloud and RGB image modalities by introducing depth encoding at both global and local levels. Specifically, the Depth-GFusion module adaptively adjusts the weights of image Bird's-Eye-View (BEV) features in multi-modal global features via depth encoding. Furthermore, to compensate for the information lost when transferring raw features to the BEV space, we propose a Depth-LFusion module, which adaptively adjusts the weights of original voxel features and multi-view image features in multi-modal local features via depth encoding. Extensive experiments on the nuScenes and KITTI datasets demonstrate that our DepthFusion method surpasses previous state-of-the-art methods. Moreover, our DepthFusion is more robust to various kinds of corruptions, outperforming previous methods on the nuScenes-C dataset.

Figures (8)

• Figure 1: Statistical analysis on the nuScenes-mini dataset. The average numbers of points and pixels for each object at different depths.
• Figure 2: Visualization analysis on the nuScenes-mini dataset. Examples of near-range and long-range objects in images and point cloud. Points within the bounding boxes are colored red for observation.
• Figure 3: Overview of our method. It introduces depth encoding in both global and local feature fusion to obtain depth-adaptive multi-modal representations for detection. is the merge operation.
• Figure 4: Illustration of the DGF.
• Figure 5: Illustration of the DLF.