Diagnosing and Predicting Autonomous Vehicle Operational Safety Using Multiple Simulation Modalities and a Virtual Environment

TL;DR

This paper addresses the safety-certification challenge for connected and automated vehicles by proposing a code-to-road hybrid testing framework that combines VIL realism with MIL and SIL efficiency. It implements a complete CTR ecosystem around a Level 3 vehicle using CARLA and a wheel-speed synchronized dynamometer to diagnose perception driven responses and to predict MIL scenarios that mimic VIL dynamics. The results show strong relationships between perception outputs and vehicle responses, and demonstrate edge-case discovery under varied weather and lighting conditions, including fog and sun angle effects. The approach enables cost effective, scalable safety evaluation and supports potential certification standards for SAE Level 2/3 autonomous driving systems.

Abstract

Even as technology and performance gains are made in the sphere of automated driving, safety concerns remain. Vehicle simulation has long been seen as a tool to overcome the cost associated with a massive amount of on-road testing for development and discovery of safety critical "edge-cases". However, purely software-based vehicle models may leave a large realism gap between their real-world counterparts in terms of dynamic response, and highly realistic vehicle-in-the-loop (VIL) simulations that encapsulate a virtual world around a physical vehicle may still be quite expensive to produce and similarly time intensive as on-road testing. In this work, we demonstrate an AV simulation test bed that combines the realism of vehicle-in-the-loop (VIL) simulation with the ease of implementation of model-in-the-loop (MIL) simulation. The setup demonstrated in this work allows for response diagnosis for the VIL simulations. By observing causal links between virtual weather and lighting conditions that surround the virtual depiction of our vehicle, the vision-based perception model and controller of Openpilot, and the dynamic response of our physical vehicle under test, we can draw conclusions regarding how the perceived environment contributed to vehicle response. Conversely, we also demonstrate response prediction for the MIL setup, where the need for a physical vehicle is not required to draw richer conclusions around the impact of environmental conditions on AV performance than could be obtained with VIL simulation alone. These combine for a simulation setup with accurate real-world implications for edge-case discovery that is both cost effective and time efficient to implement.

Figures (8)

• Figure 1: An overview of the Vehicle-in-the-loop(VIL), Software-in-the-loop(SIL) and Model-in-the-loop(MIL) simulations used in this work. In VIL, the feedback to the simulator is vehicle kinematics measured from the CAN bus of the vehicle directly. In SIL, the controller feeds control commands to the simulator directly, and the physics model within CARLA determines the dynamic response. In MIL, the CARLA vehicle follows a predetermined trajectory, feeding images to the Openpilot perception model.
• Figure 2: The complete VIL Simulation setup is shown. (Left) The 2019 Toyota RAV4 is shown on the chassis dynamometer. The device that allows for steering and synchronized wheel speed is attached. (Right) The kinematic model used to update the position of the vehicle in CARLA using the dynamic response of the vehicle is shown. The wheelbase $L$, the heading $\theta$, and the steering angle $\delta$ are illustrated, the heading and the position of the vehicle $(x,y,\theta)$ are defined in terms of arbitrary world coordinates defined by the CARLA simulator.
• Figure 3: Driving types and the primary weather and lighting effects under test are shown. For the two types of driving on the right, the ego vehicle is shown in a blue box, the lead vehicle is shown in an orange box, and the driving path under test is shown as a dashed red line. On the right, an image was captured 8 seconds into the Stopping test for clear weather, rain, sun glare, and night type driving with headlights.
• Figure 4: Performance metrics are shown. Bar length represents the mean score for all runs, with number of runs shown in \ref{['nruns']}. Vertical black bars indicate the minimum and maximum measured scores for each metric.
• Figure 5: Raw data for the stopping experiments is shown. \ref{['rawfig:control-accel']} shows the control acceleration signal supplied to the car during the deceleration curve plotted against lead distance $d_{t}$. \ref{['rawfig:obrat']} plots the detection ratio for the Stopping experiments in the same distance range. Actual trajectory samples are shown as transparent, while the dotted lines indicate the mean value for each experiment.