Is Your HD Map Constructor Reliable under Sensor Corruptions?

TL;DR

This work introduces MapBench, the first comprehensive benchmark designed to evaluate the robustness of HD map construction methods against various sensor corruptions, and identifies effective strategies for enhancing robustness, including innovative approaches that leverage multi-modal fusion, advanced data augmentation, and architectural techniques.

Abstract

Driving systems often rely on high-definition (HD) maps for precise environmental information, which is crucial for planning and navigation. While current HD map constructors perform well under ideal conditions, their resilience to real-world challenges, \eg, adverse weather and sensor failures, is not well understood, raising safety concerns. This work introduces MapBench, the first comprehensive benchmark designed to evaluate the robustness of HD map construction methods against various sensor corruptions. Our benchmark encompasses a total of 29 types of corruptions that occur from cameras and LiDAR sensors. Extensive evaluations across 31 HD map constructors reveal significant performance degradation of existing methods under adverse weather conditions and sensor failures, underscoring critical safety concerns. We identify effective strategies for enhancing robustness, including innovative approaches that leverage multi-modal fusion, advanced data augmentation, and architectural techniques. These insights provide a pathway for developing more reliable HD map construction methods, which are essential for the advancement of autonomous driving technology. The benchmark toolkit and affiliated code and model checkpoints have been made publicly accessible.

Figures (17)

• Figure 1: Radar charts of state-of-the-art HD map constructors under the Camera and LiDAR sensor corruptions. We report the mAP scores of different map construction methods under each corruption type across severity levels. Camera Corruptions:#1Clean, #2Frame Lost, #3Camera Crash, #4Low-Light, #5Bright, #6Color Quant, #7Snow, #8Fog, and #9Motion Blur. LiDAR Corruptions:#1Clean, #2Wet Ground, #3Snow, #4Motion Blur, #5Incomplete Echo, #6Fog, #7Crosstalk, #8Cross-Sensor, and #9Beam Missing. The radius of each chart is normalized based on the Clean score. The larger the area coverage, the better the overall robustness.
• Figure 2: Definitions of the Camera and LiDAR sensor corruptions in MapBench. Our benchmark encompasses a total of 16 corruption types for HD map construction, which can be categorized into exterior, interior, and sensor failure scenarios. Besides, we define 13 multi-sensor corruptions by combining the camera and LiDAR sensor failure types. Kindly refer to our Appendix for more details.
• Figure 3: The correlations of accuracy (mAP) and robustness (mCE / mRR) for the Camera (a) and (b) and LiDAR (c) and (d) models. The size of the circle represents the number of model parameters.
• Figure 4: The results of Camera-LiDAR fusion methods MapTRHIMap under multi-sensor corruptions.
• Figure 5: Qualitative assessment of camera-LiDAR fusion-based HD map construction under the Camera and LiDAR combined sensor corruptions. Kindly refer to Sec. \ref{['sec:supp_qualitative']} for additional examples.