MSC-Bench: Benchmarking and Analyzing Multi-Sensor Corruption for Driving Perception

TL;DR

This work introduces MSC-Bench, the first comprehensive benchmark for evaluating the robustness of multi-sensor driving perception under 16 corruption types. It targets two key tasks—3D object detection and HD map construction—evaluating six detectors and four map constructors with new resilience metrics RS, mRS, RRS, and mRRS. The findings reveal that dual-source disruptions markedly degrade performance and that robustness is not reliably predicted by clean-set accuracy, underscoring the need for fault-tolerant fusion methods. The benchmark and code are released to enable reproducibility and drive development of more dependable multi-sensor perception systems for autonomous driving.

Abstract

Multi-sensor fusion models play a crucial role in autonomous driving perception, particularly in tasks like 3D object detection and HD map construction. These models provide essential and comprehensive static environmental information for autonomous driving systems. While camera-LiDAR fusion methods have shown promising results by integrating data from both modalities, they often depend on complete sensor inputs. This reliance can lead to low robustness and potential failures when sensors are corrupted or missing, raising significant safety concerns. To tackle this challenge, we introduce the Multi-Sensor Corruption Benchmark (MSC-Bench), the first comprehensive benchmark aimed at evaluating the robustness of multi-sensor autonomous driving perception models against various sensor corruptions. Our benchmark includes 16 combinations of corruption types that disrupt both camera and LiDAR inputs, either individually or concurrently. Extensive evaluations of six 3D object detection models and four HD map construction models reveal substantial performance degradation under adverse weather conditions and sensor failures, underscoring critical safety issues. The benchmark toolkit and affiliated code and model checkpoints have been made publicly accessible.

Figures (6)

• Figure 1: Radar charts display the performance of state-of-the-art multi-sensor 3D object detection models (left) and HD map construction models (right) under the Multi-Sensor Corruption Benchmark (MSC-Bench). We present NDS scores for 3D object detection methods and mAP scores for map construction methods across each corruption type and severity level. MSC-Bench:#1Clean, #2Motion Blur, #3Temporal Misalignment, #4Spatial Misalignment, #5Fog, #6Snow, #7Camera Crash, #8Frame Lost, #9Cross Sensor, #10Cross Talk, #11Incomplete Echo, #12Camera Crash $\&$ Cross Sensor, #13Camera Crash $\&$ Cross Talk, #14Camera Crash $\&$ Incomplete Echo, #15Frame Lost $\&$ Cross Sensor, #16Frame Lost $\&$ Cross Talk and #17Frame Lost $\&$ Incomplete Echo. The radius of each chart is normalized based on the Clean score. The larger the area coverage, the better the overall robustness.
• Figure 2: Overview of the MSC-Bench. Definitions of the multi-sensor corruptions in MSC-Bench. Our benchmark encompasses a total of 16 corruption types for multi-modal perception models, which can be categorized into weather, interior, and sensor failure scenarios.
• Figure 3: Robustness against all corruption types and severity levels in 3D object detection tasks is evaluated through the Resilience Score (RS), calculated using the NDS score for varying severity levels.
• Figure 4: Robustness against all corruption types and severity levels in HD map construction tasks is assessed using the Resilience Score (RS), calculated based on the mAP score for different severity levels.
• Figure 5: Relative robustness visualization. Relative Resilience Score (RRS) computed with NDS using BEVFusion liu2023bevfusion as baseline.