Analyzing the Impact of Simulation Fidelity on the Evaluation of Autonomous Driving Motion Control

TL;DR

This paper tackles the problem of disparate vehicle dynamics fidelity in autonomous driving simulations by constructing a high-fidelity Autoware-compatible baseline model and multiple simplified variants, then evaluating their closed-loop impact on trajectory-following using Indy Monza 2023 data. The authors define $d_{ ext{max}}$ and $d_{ ext{dis},i}$ to quantify both peak and curve-consistency performance and conduct 30-repeat reproducibility experiments plus scaled-velocity tests across models. They find that a carefully parameterized baseline or modest variations (e.g., tire parameters) closely match real-world performance, while aggressive simplifications substantially degrade closed-loop fidelity, especially under high lateral accelerations. The work provides practical guidance on selecting appropriate model fidelity for simulation-based evaluation and offers an open-source, Autoware-compatible vehicle model to enable reproducible cross-algorithm comparisons.

Abstract

Simulation is crucial in the development of autonomous driving software. In particular, assessing control algorithms requires an accurate vehicle dynamics simulation. However, recent publications use models with varying levels of detail. This disparity makes it difficult to compare individual control algorithms. Therefore, this paper aims to investigate the influence of the fidelity of vehicle dynamics modeling on the closed-loop behavior of trajectory-following controllers. For this purpose, we introduce a comprehensive Autoware-compatible vehicle model. By simplifying this, we derive models with varying fidelity. Evaluating over 550 simulation runs allows us to quantify each model's approximation quality compared to real-world data. Furthermore, we investigate whether the influence of model simplifications changes with varying margins to the acceleration limit of the vehicle. From this, we deduce to which degree a vehicle model can be simplified to evaluate control algorithms depending on the specific application. The real-world data used to validate the simulation environment originate from the Indy Autonomous Challenge race at the Autodromo Nazionale di Monza in June 2023. They show the fastest fully autonomous lap of TUM Autonomous Motorsport, with vehicle speeds reaching 267 kph and lateral accelerations of up to 15 mps2.

Figures (7)

• Figure 1: The Dallara AV-21 race car used for data recording on the Autodromo Nazionale di Monza in June 2023.
• Figure 2: Simulation architecture used in this paper. Blue coloring indicates a component of the in-loop autonomous driving software. Each block corresponds to one ROS 2 node.
• Figure 3: Lateral displacement error $d$ of the tracking control algorithm shown for the recorded real-world data and in simulation. In addition to the model developed during this work $\mathcal{V}_{\mathrm{base}}$, the plot shows another compatible open-source vehicle model hermansdorfer2022 for comparison.
• Figure 4: Reproducibility regarding the analyzed metrics.
• Figure 5: Maximum lateral tracking error for each simulation model during the reference lap. The real-world data shown for comparison represent a single sample.