S2R-Bench: A Sim-to-Real Evaluation Benchmark for Autonomous Driving

TL;DR

S2R-Bench provides a comprehensive sim-to-real evaluation framework for autonomous driving perception by pairing real-world sensor anomaly data with corresponding simulated corruptions across camera, LiDAR, and 4D radar. It encompasses diverse road types, lighting, and adverse weather, with three integrated datasets (S2R-C, S2R-R, S2R-S) and a KITTI-formatted release to enable cross-modality, cross-condition benchmarking. The paper demonstrates baseline performance across unimodal and multimodal methods and analyzes the sim-to-real gap to guide robust perception development. This benchmark advances reliable perception under real-world disturbances and supports downstream tasks such as anomaly detection and sensor fusion for safer autonomous driving deployment.

Abstract

Safety is a long-standing and the final pursuit in the development of autonomous driving systems, with a significant portion of safety challenge arising from perception. How to effectively evaluate the safety as well as the reliability of perception algorithms is becoming an emerging issue. Despite its critical importance, existing perception methods exhibit a limitation in their robustness, primarily due to the use of benchmarks are entierly simulated, which fail to align predicted results with actual outcomes, particularly under extreme weather conditions and sensor anomalies that are prevalent in real-world scenarios. To fill this gap, in this study, we propose a Sim-to-Real Evaluation Benchmark for Autonomous Driving (S2R-Bench). We collect diverse sensor anomaly data under various road conditions to evaluate the robustness of autonomous driving perception methods in a comprehensive and realistic manner. This is the first corruption robustness benchmark based on real-world scenarios, encompassing various road conditions, weather conditions, lighting intensities, and time periods. By comparing real-world data with simulated data, we demonstrate the reliability and practical significance of the collected data for real-world applications. We hope that this dataset will advance future research and contribute to the development of more robust perception models for autonomous driving. This dataset is released on https://github.com/adept-thu/S2R-Bench.

Figures (6)

• Figure 1: The impact of diverse adverse weather conditions and driving scenarios on camera lens clarity and captured image quality in autonomous driving environments.
• Figure 2: The configuration of our experiment platform and visualisation scenarios. (a) Shows the self-driving car system as well as the real anomaly data collected and the simulated anomaly data. (b) shows the statistics of the number of frames for various periods in different scenarios. (c) shows the statistics of the number of objects in different labels. The result suggests that the main labels like Car, Pedestrian, and Cyclist take up over three-quarters of the total amount of objects.
• Figure 3: Projection visualization of sensor calibration. (a), (b), (c), and (d) represent the projection of the calibrated data (3D bounding box, LiDAR point cloud, Arbe Phoenix point cloud, and Oculli-Eagle point cloud) on the image.
• Figure 4: The statistic of different annotated objects at different ranges of distance from the ego vehicle. From the results, the majority of the annotated objects are in the range of 20m-80m.
• Figure 5: Representing 3D annotations in multiple scenarios and sensor modalities. The four columns respectively display the projection of 3D annotation boxes in images, LiDAR point clouds, Arbe Phoenix and Oculli-Eagle radar point clouds. Each row represents a scenario type. (a) light snow $\&$ real; (b) light snow $\&$ simulation; (c) moderate snow $\&$ real; (d) moderate snow $\&$ simulation; (e) fog $\&$ real; (f) fog $\&$ simulation; (g) brightness $\&$ real; (h) brightness $\&$ simulation.