Simulation-Based Performance Evaluation of 3D Object Detection Methods with Deep Learning for a LiDAR Point Cloud Dataset in a SOTIF-related Use Case

TL;DR

This paper tackles how Safety of the Intended Functionality (SOTIF) constraints influence LiDAR-based 3D object detection for automated driving. It proposes a methodology that uses the CARLA simulator to generate a KITTI-format LiDAR dataset across 21 weather scenarios (547 frames) and evaluates pre-trained DL detectors from MMDetection3D and OpenPCDet on this data, using AP$_{11}$/AP$_{40}$ at IoU $0.70$ and Recall at IoU thresholds. The study reports that detectors like PV-RCNN and PointPillars perform strongly in easy settings, while OpenPCDet recalls remain solid at IoU $0.30$ but drop at $0.50$, illustrating a domain gap between simulated and real data. The findings demonstrate the feasibility of applying pre-trained models to simulation-derived datasets and underscore the need for targeted adaptation and future directions in uncertainty quantification to enhance reliability in SOTIF-relevant scenarios.

Abstract

Safety of the Intended Functionality (SOTIF) addresses sensor performance limitations and deep learning-based object detection insufficiencies to ensure the intended functionality of Automated Driving Systems (ADS). This paper presents a methodology examining the adaptability and performance evaluation of the 3D object detection methods on a LiDAR point cloud dataset generated by simulating a SOTIF-related Use Case. The major contributions of this paper include defining and modelling a SOTIF-related Use Case with 21 diverse weather conditions and generating a LiDAR point cloud dataset suitable for application of 3D object detection methods. The dataset consists of 547 frames, encompassing clear, cloudy, rainy weather conditions, corresponding to different times of the day, including noon, sunset, and night. Employing MMDetection3D and OpenPCDET toolkits, the performance of State-of-the-Art (SOTA) 3D object detection methods is evaluated and compared by testing the pre-trained Deep Learning (DL) models on the generated dataset using Average Precision (AP) and Recall metrics.

Figures (10)

• Figure 1: Schematic Representation of the Methodological Approach
• Figure 2: Description of the sotif-related Use Case
• Figure 3: Generated dataset structure
• Figure 4: CARLA Simulation Environment Framework
• Figure 5: Overview of point cloud-based object detection