AgentDrive: An Open Benchmark Dataset for Agentic AI Reasoning with LLM-Generated Scenarios in Autonomous Systems

TL;DR

AgentDrive addresses the lack of scalable, safety-critical benchmarks for agentic AI in autonomous driving by introducing a fully generative, simulation-grounded benchmark built on a seven-axis scenario space. The framework comprises AgentDrive-Gen for generating 300k structured driving scenarios, AgentDrive-Sim for simulation rollouts and labeling, and AgentDrive-MCQ for reasoning-focused evaluation across physics, policy, hybrid, scenario, and comparative dimensions. A large-scale evaluation of 50 LLMs reveals that frontier proprietary models excel in contextual and policy reasoning, while advanced open models are rapidly closing gaps in structured and physics-grounded reasoning, highlighting the role of model scale, training strategies, and alignment. The dataset and evaluation code are openly released, enabling training, fine-tuning, and robust benchmarking for safety-aware, reasoning-driven autonomous agents and guiding future multimodal and multi-agent extensions.

Abstract

The rapid advancement of large language models (LLMs) has sparked growing interest in their integration into autonomous systems for reasoning-driven perception, planning, and decision-making. However, evaluating and training such agentic AI models remains challenging due to the lack of large-scale, structured, and safety-critical benchmarks. This paper introduces AgentDrive, an open benchmark dataset containing 300,000 LLM-generated driving scenarios designed for training, fine-tuning, and evaluating autonomous agents under diverse conditions. AgentDrive formalizes a factorized scenario space across seven orthogonal axes: scenario type, driver behavior, environment, road layout, objective, difficulty, and traffic density. An LLM-driven prompt-to-JSON pipeline generates semantically rich, simulation-ready specifications that are validated against physical and schema constraints. Each scenario undergoes simulation rollouts, surrogate safety metric computation, and rule-based outcome labeling. To complement simulation-based evaluation, we introduce AgentDrive-MCQ, a 100,000-question multiple-choice benchmark spanning five reasoning dimensions: physics, policy, hybrid, scenario, and comparative reasoning. We conduct a large-scale evaluation of fifty leading LLMs on AgentDrive-MCQ. Results show that while proprietary frontier models perform best in contextual and policy reasoning, advanced open models are rapidly closing the gap in structured and physics-grounded reasoning. We release the AgentDrive dataset, AgentDrive-MCQ benchmark, evaluation code, and related materials at https://github.com/maferrag/AgentDrive

Figures (6)

• Figure 1: Overview of the AgentDrive benchmark suite, which comprises three complementary datasets: AgentDrive-Gen for LLM-based driving scenario generation, AgentDrive-Sim for simulation and outcome labeling, and AgentDrive-MCQ for reasoning and decision-making evaluation.
• Figure 2: AgentDrive Framework: End-to-End Pipeline for LLM-Generated Autonomous Driving Scenarios.
• Figure 3: Distributions and layout statistics across the AgentDrive scenario dataset, showing difficulty levels, layout correlations, temporal and environmental characteristics.
• Figure 4: Label distribution of AgentDrive-Sim across episode and segment levels.
• Figure 5: The AgentDrive-MCQ framework.