$α^3$-Bench: A Unified Benchmark of Safety, Robustness, and Efficiency for LLM-Based UAV Agents over 6G Networks

TL;DR

α^3-Bench tackles the challenge of evaluating LLM-driven UAV autonomy under realistic 6G networking by formulating missions as multi-turn, language-mediated control loops. It introduces a unified conversational decision framework with MCP and A2A protocols, a 6G-aware network context, and a composite α^3 metric that combines six pillars—Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost—with reliability and efficiency normalization. The paper builds a large-scale corpus of 113k AI conversational UAV episodes grounded in UAVBench, evaluating 17 state-of-the-art LLMs on a fixed subset of 50 episodes per scenario. Experimental results reveal that while many models achieve high mission success and safety, robustness to degraded 6G conditions and efficiency vary significantly, underscoring the importance of network-aware and resource-efficient UAV agents. The work provides a reproducible benchmark with public data and metrics, establishing a principled foundation for evaluating trustworthy conversational AI in networked autonomous systems and outlining future extensions to multimodal perception and broader 6G-enabled domains.

Abstract

Large Language Models (LLMs) are increasingly used as high level controllers for autonomous Unmanned Aerial Vehicle (UAV) missions. However, existing evaluations rarely assess whether such agents remain safe, protocol compliant, and effective under realistic next generation networking constraints. This paper introduces $α^3$-Bench, a benchmark for evaluating LLM driven UAV autonomy as a multi turn conversational reasoning and control problem operating under dynamic 6G conditions. Each mission is formulated as a language mediated control loop between an LLM based UAV agent and a human operator, where decisions must satisfy strict schema validity, mission policies, speaker alternation, and safety constraints while adapting to fluctuating network slices, latency, jitter, packet loss, throughput, and edge load variations. To reflect modern agentic workflows, $α^3$-Bench integrates a dual action layer supporting both tool calls and agent to agent coordination, enabling evaluation of tool use consistency and multi agent interactions. We construct a large scale corpus of 113k conversational UAV episodes grounded in UAVBench scenarios and evaluate 17 state of the art LLMs using a fixed subset of 50 episodes per scenario under deterministic decoding. We propose a composite $α^3$ metric that unifies six pillars: Task Outcome, Safety Policy, Tool Consistency, Interaction Quality, Network Robustness, and Communication Cost, with efficiency normalized scores per second and per thousand tokens. Results show that while several models achieve high mission success and safety compliance, robustness and efficiency vary significantly under degraded 6G conditions, highlighting the need for network aware and resource efficient LLM based UAV agents. The dataset is publicly available on GitHub : https://github.com/maferrag/AlphaBench

Figures (7)

• Figure 1: End-to-end workflow of the $\alpha^3$-Bench framework for evaluating LLM-based UAV agents under dynamic 6G communication conditions. The figure illustrates scenario initialization from UAVBench, dialogue-based mission execution with network-aware reasoning, structured action invocation via MCP and A2A protocols, environment and state updates, loop termination, and final efficiency- and reliability-adjusted $\alpha^3$ evaluation metrics.
• Figure 2: An example agent--user interaction trajectory in the UAV domain of $\alpha^3$-Bench under 6G communication. The left panel illustrates user--agent interactions via the Model Context Protocol (MCP) modelcontextprotocol2025, where the UAV state is queried and a thermal area-scan mission is initiated over a 6G eMBB slice. The right panel highlights the agent’s adaptive decision-making under dynamic network conditions, including coordination with other autonomous agents through the agent-to-agent (A2A) googleblog2025 protocol for collision avoidance, and seamless switching between eMBB, URLLC, and mMTC 6G network slices to preserve safety and mission continuity.
• Figure 3: Overall performance comparison of LLM agents under the $\alpha^{3}$-Bench.
• Figure 4: Reliability, coverage, and success rate across LLM models.
• Figure 5: Generation failure rate per LLM model.