Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UAVBench: An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios

Mohamed Amine Ferrag, Abderrahmane Lakas, Merouane Debbah

TL;DR

This work targets the gap in physically grounded benchmarks for autonomous UAVs guided by LLMs. It introduces UAVBench, a large-scale open dataset of $50{,}000$ validated flight scenarios encoded in a structured JSON schema, generated via taxonomy-guided LLM prompting and a multi-stage validation and risk-labeling pipeline. It further extends this foundation with UAVBench_MCQ, a reasoning-focused benchmark containing $50{,}000$ MCQs across ten reasoning styles to enable interpretable and machine-checkable evaluation of UAV cognition. A comprehensive evaluation of $32$ state-of-the-art LLMs reveals strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making, particularly in multi-agent coordination and energy management. The work provides a reproducible platform for benchmarking and advancing agentic UAV intelligence, with public release on GitHub to foster open science and reproducibility.

Abstract

Autonomous aerial systems increasingly rely on large language models (LLMs) for mission planning, perception, and decision-making, yet the lack of standardized and physically grounded benchmarks limits systematic evaluation of their reasoning capabilities. To address this gap, we introduce UAVBench, an open benchmark dataset comprising 50,000 validated UAV flight scenarios generated through taxonomy-guided LLM prompting and multi-stage safety validation. Each scenario is encoded in a structured JSON schema that includes mission objectives, vehicle configuration, environmental conditions, and quantitative risk labels, providing a unified representation of UAV operations across diverse domains. Building on this foundation, we present UAVBench_MCQ, a reasoning-oriented extension containing 50,000 multiple-choice questions spanning ten cognitive and ethical reasoning styles, ranging from aerodynamics and navigation to multi-agent coordination and integrated reasoning. This framework enables interpretable and machine-checkable assessment of UAV-specific cognition under realistic operational contexts. We evaluate 32 state-of-the-art LLMs, including GPT-5, ChatGPT-4o, Gemini 2.5 Flash, DeepSeek V3, Qwen3 235B, and ERNIE 4.5 300B, and find strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making. UAVBench establishes a reproducible and physically grounded foundation for benchmarking agentic AI in autonomous aerial systems and advancing next-generation UAV reasoning intelligence. To support open science and reproducibility, we release the UAVBench dataset, the UAVBench_MCQ benchmark, evaluation scripts, and all related materials on GitHub at https://github.com/maferrag/UAVBench

UAVBench: An Open Benchmark Dataset for Autonomous and Agentic AI UAV Systems via LLM-Generated Flight Scenarios

TL;DR

This work targets the gap in physically grounded benchmarks for autonomous UAVs guided by LLMs. It introduces UAVBench, a large-scale open dataset of validated flight scenarios encoded in a structured JSON schema, generated via taxonomy-guided LLM prompting and a multi-stage validation and risk-labeling pipeline. It further extends this foundation with UAVBench_MCQ, a reasoning-focused benchmark containing MCQs across ten reasoning styles to enable interpretable and machine-checkable evaluation of UAV cognition. A comprehensive evaluation of state-of-the-art LLMs reveals strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making, particularly in multi-agent coordination and energy management. The work provides a reproducible platform for benchmarking and advancing agentic UAV intelligence, with public release on GitHub to foster open science and reproducibility.

Abstract

Autonomous aerial systems increasingly rely on large language models (LLMs) for mission planning, perception, and decision-making, yet the lack of standardized and physically grounded benchmarks limits systematic evaluation of their reasoning capabilities. To address this gap, we introduce UAVBench, an open benchmark dataset comprising 50,000 validated UAV flight scenarios generated through taxonomy-guided LLM prompting and multi-stage safety validation. Each scenario is encoded in a structured JSON schema that includes mission objectives, vehicle configuration, environmental conditions, and quantitative risk labels, providing a unified representation of UAV operations across diverse domains. Building on this foundation, we present UAVBench_MCQ, a reasoning-oriented extension containing 50,000 multiple-choice questions spanning ten cognitive and ethical reasoning styles, ranging from aerodynamics and navigation to multi-agent coordination and integrated reasoning. This framework enables interpretable and machine-checkable assessment of UAV-specific cognition under realistic operational contexts. We evaluate 32 state-of-the-art LLMs, including GPT-5, ChatGPT-4o, Gemini 2.5 Flash, DeepSeek V3, Qwen3 235B, and ERNIE 4.5 300B, and find strong performance in perception and policy reasoning but persistent challenges in ethics-aware and resource-constrained decision-making. UAVBench establishes a reproducible and physically grounded foundation for benchmarking agentic AI in autonomous aerial systems and advancing next-generation UAV reasoning intelligence. To support open science and reproducibility, we release the UAVBench dataset, the UAVBench_MCQ benchmark, evaluation scripts, and all related materials on GitHub at https://github.com/maferrag/UAVBench

Paper Structure

This paper contains 42 sections, 26 equations, 5 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Vision-Language Navigation Platforms for UAVs
  4. Embodied Aerospace Intelligence
  5. Benchmarks for UAV-Embodied Agents
  6. Object Goal Navigation in UAVs
  7. Embodied Cognition in Urban Airspaces
  8. Comparative Analysis of UAV Benchmarks
  9. UAVBench - Dataset Generation Methodology
  10. Scenario Schema Design
  11. Simulation Setup
  12. UAV Configuration and Propulsion
  13. Payload Taxonomy and Examples
  14. Environment and Airspace
  15. Mission and Entities
  16. ...and 27 more sections

Figures (5)

  • Figure 1: UAVBench Dataset Generation, Validation, and Labeling Framework.
  • Figure 2: Overview of UAVBench dataset composition and UAV design characteristics. (a) Mission types illustrating the diversity of operational scenarios; (b) Airspace types showing the range of environmental contexts; (c) Weather phenomena highlighting atmospheric complexity; (d) UAV mass distribution indicating variability in platform sizes; (e) Battery capacity distribution reflecting energy endurance profiles; and (f) Payload category frequencies summarizing the variety of onboard sensors and mission payloads.
  • Figure 3: Overview of UAVBench_MCQ dataset structure and linguistic statistics. (a) Distribution of multiple-choice question (MCQ) styles across reasoning domains; (b) Number of answer choices per question; (c) Distribution of question lengths in words; (d) Comparison of word counts for questions, averaged choices, and rationales; (e) Average choice length by option label (A–G); and (f) The most frequent starting verbs in the choice text. Together, these subfigures summarize the content balance, linguistic complexity, and stylistic diversity of UAVBench_MCQ items.
  • Figure 4: UAVBench_MCQ Creation Pipeline.
  • Figure 5: Top 15 UAVBench_MCQ models ranked by Balanced Style Score (BSS). (a) Mean accuracy across ten reasoning styles, (b) cross-style consistency measured as the standard deviation of accuracies (lower is better; axis in %), and (c) BSS integrating both accuracy and consistency.