PEM: Perception Error Model for Virtual Testing of Autonomous Vehicles

TL;DR

This article defines Perception Error Models (PEM), a virtual simulation component that can enable the analysis of the impact of perception errors on AV safety, without the need to model the sensors themselves.

Abstract

Even though virtual testing of Autonomous Vehicles (AVs) has been well recognized as essential for safety assessment, AV simulators are still undergoing active development. One particularly challenging question is to effectively include the Sensing and Perception (S&P) subsystem into the simulation loop. In this article, we define Perception Error Models (PEM), a virtual simulation component that can enable the analysis of the impact of perception errors on AV safety, without the need to model the sensors themselves. We propose a generalized data-driven procedure towards parametric modeling and evaluate it using Apollo, an open-source driving software, and nuScenes, a public AV dataset. Additionally, we implement PEMs in SVL, an open-source vehicle simulator. Furthermore, we demonstrate the usefulness of PEM-based virtual tests, by evaluating camera, LiDAR, and camera-LiDAR setups. Our virtual tests highlight limitations in the current evaluation metrics, and the proposed approach can help study the impact of perception errors on AV safety.

Figures (10)

• Figure 1: Comparison of three different testing setups that integrate perception: physical testing (top), synthetic signals in a virtual environment (middle), and our proposed approach involving PEMs (bottom), models that approximate the error of a Sensing and Perception module.
• Figure 2: Illustration of critical issues I1, I2, I3. We indicate the ego vehicles in blue, other objects in green, and detections in red. (I1) Temporal Relevance: The detection timeline of the vehicle in lane 1 is more sporadic and unstable, but non-detection intervals are shorter than the vehicle in lane 2. (I2): Overlap Sensitivity: A detection bounding box can be closer, farther, or on the side relative to the actual vehicle. (I3) Relevance of the objects: Many objects are present. In each of these examples, the context of the error affects its severity.
• Figure 3: Software pipeline to generate $\mathcal{\tilde{W}}$ by means of Apollo Fan2018 and nuScenes Caesar2020.
• Figure 4: Example of a zone-based partitioning of PEM. A polar grid can meaningfully organize the surrounding area into zones, which different sensors may cover with a different degree of reliability.
• Figure 5: Illustration of 2 steps (frames) of PEM we deploy in our experiments.