A Tool for Benchmarking Large Language Models' Robustness in Assessing the Realism of Driving Scenarios
Jiahui Wu, Chengjie Lu, Aitor Arrieta, Shaukat Ali
TL;DR
The paper addresses the challenge of validating the realism of simulation-based driving scenarios for ADS safety. It proposes DriveRLR, a benchmark that mutates driving scenarios, builds prompts, and uses LLMs to judge realism, enabling robustness analysis across variant scenarios. Experiments on the DeepScenario dataset with GPT-5, Llama 4 Maverick, and Mistral Small 3.2 demonstrate DriveRLR’s ability to reveal cross-model differences in realism assessment and to function as an objective function for scenario generation. The approach is open-source and designed to integrate into ADS testing pipelines, offering practical utility for targeted realism evaluation and scenario curation.
Abstract
In recent years, autonomous driving systems have made significant progress, yet ensuring their safety remains a key challenge. To this end, scenario-based testing offers a practical solution, and simulation-based methods have gained traction due to the high cost and risk of real-world testing. However, evaluating the realism of simulated scenarios remains difficult, creating demand for effective assessment methods. Recent advances show that Large Language Models (LLMs) possess strong reasoning and generalization capabilities, suggesting their potential in assessing scenario realism through scenario-related textual prompts. Motivated by this, we propose DriveRLR, a benchmark tool to assess the robustness of LLMs in evaluating the realism of driving scenarios. DriveRLR generates mutated scenario variants, constructs prompts, which are then used to assess a given LLM's ability and robustness in determining the realism of driving scenarios. We validate DriveRLR on the DeepScenario dataset using three state-of-the-art LLMs: GPT-5, Llama 4 Maverick, and Mistral Small 3.2. Results show that DriveRLR effectively reveals differences in the robustness of various LLMs, demonstrating its effectiveness and practical value in scenario realism assessment. Beyond LLM robustness evaluation, DriveRLR can serve as a practical component in applications such as an objective function to guide scenario generation, supporting simulation-based ADS testing workflows.