LCF3D: A Robust and Real-Time Late-Cascade Fusion Framework for 3D Object Detection in Autonomous Driving

TL;DR

LCF3D proposes a robust late-cascade fusion framework for autonomous-driving 3D object detection by combining a 2D RGB detector with a 3D LiDAR detector. It introduces three modules—Bounding Box Matching, Detection Recovery, and Semantic Fusion—to suppress LiDAR false positives, recover missed detections, and harmonize labels across modalities, all while enabling parallel processing for real-time performance. The method demonstrates strong domain generalization across sensor configurations on KITTI and nuScenes, notably boosting detection of small and distant objects such as Pedestrians and Cyclists, and Bicycles/Motorcycles, while remaining detector-agnostic. The approach achieves competitive or superior results compared with LiDAR-only and multimodal baselines, with a favorable latency/accuracy trade-off and practical applicability to real-world autonomous driving.

Abstract

Accurately localizing 3D objects like pedestrians, cyclists, and other vehicles is essential in Autonomous Driving. To ensure high detection performance, Autonomous Vehicles complement RGB cameras with LiDAR sensors, but effectively combining these data sources for 3D object detection remains challenging. We propose LCF3D, a novel sensor fusion framework that combines a 2D object detector on RGB images with a 3D object detector on LiDAR point clouds. By leveraging multimodal fusion principles, we compensate for inaccuracies in the LiDAR object detection network. Our solution combines two key principles: (i) late fusion, to reduce LiDAR False Positives by matching LiDAR 3D detections with RGB 2D detections and filtering out unmatched LiDAR detections; and (ii) cascade fusion, to recover missed objects from LiDAR by generating new 3D frustum proposals corresponding to unmatched RGB detections. Experiments show that LCF3D is beneficial for domain generalization, as it turns out to be successful in handling different sensor configurations between training and testing domains. LCF3D achieves significant improvements over LiDAR-based methods, particularly for challenging categories like pedestrians and cyclists in the KITTI dataset, as well as motorcycles and bicycles in nuScenes. Code can be downloaded from: https://github.com/CarloSgaravatti/LCF3D.

Figures (12)

• Figure 1: LCF3D consists of two parallel branches and three sequential steps. The RGB branch (a) produces 2D detections $\mathcal{D}^{2D}$, the LiDAR branch (b) generates 3D detections $\widehat{\mathcal{D}}^{3D}$ from the point cloud ${\mathcal{P}}$. In step (c), 3D detections are projected and matched with 2D ones (${\mathcal{M}}$). Unmatched RGB detections ${\mathcal{U}}$ are processed in step (d) by the Detection Recovery module, which uses Frustum Proposals and a Frustum Localizer to recover missed LiDAR detections (${\mathcal{R}}$). Step (e) employs Semantic Fusion to enforce consistency between LiDAR and RGB branches.
• Figure 2: Comparison between the LiDAR branch output (left) and the Bbox Matching module output (right), which removes FPs. Only the highest-confidence bounding box per cluster is shown on the left.
• Figure 3: High-level overview of the Bounding Box Matching for one image. First, 3D boxes are clustered in BEV, projected onto the image, and matched with 2D detections using IoU. Finally, each 3D cluster is matched with the 2D detections using an optimization problem based on the IoU and the bounding box with the highest score inside each matched cluster is selected (Cluster-wise NMS).
• Figure 4: (Left) Projection of all the 3D bounding box of a cluster $Z_c$ onto the image plane. (Right) Comparison with RGB 2D detections (blue): green boxes have IoU $> 0.5$, red boxes are below the threshold. The maximum IoU in the cluster is used to resolve conflicts.
• Figure 5: In the single-view scenario, the Detection Recovery module processes an unmatched 2D RGB bounding box $\mathbf{b}_{j}$. A Frustum Proposal ${\mathcal{P}}_{j}$ is generated by projecting the 3D points of the input Point Cloud ${\mathcal{P}}$ into the image and selecting points within $\mathbf{b}_{j}$. The Frustum Localizer then extracts a 3D bounding box $\mathbf{B}_{j}$ from the frustum, which inherits the semantic label of the 2D detection.