$α^3$-SecBench: A Large-Scale Evaluation Suite of Security, Resilience, and Trust for LLM-based UAV Agents over 6G Networks
Mohamed Amine Ferrag, Abderrahmane Lakas, Merouane Debbah
TL;DR
α^3-SecBench introduces a large-scale, episode-centric security benchmark for evaluating LLM-based UAV agents in 6G environments. By embedding 20,000 validated security overlays across 7 autonomy layers and 175 threat types into 113,475 baseline UAV missions, the framework enables reproducible assessment of security, resilience, and trust without altering agent internals. Across 23 state-of-the-art LLMs, results reveal robust attack detection but inconsistent mitigation, with CWE attribution lagging behind detection and safe-degradation actions often incomplete. The work provides a principled scoring scheme, CWE-aware evaluation, and a public repository, offering a path toward deployment-ready, secure autonomous UAV systems under adversarial conditions in next-generation networks.
Abstract
Autonomous unmanned aerial vehicle (UAV) systems are increasingly deployed in safety-critical, networked environments where they must operate reliably in the presence of malicious adversaries. While recent benchmarks have evaluated large language model (LLM)-based UAV agents in reasoning, navigation, and efficiency, systematic assessment of security, resilience, and trust under adversarial conditions remains largely unexplored, particularly in emerging 6G-enabled settings. We introduce $α^{3}$-SecBench, the first large-scale evaluation suite for assessing the security-aware autonomy of LLM-based UAV agents under realistic adversarial interference. Building on multi-turn conversational UAV missions from $α^{3}$-Bench, the framework augments benign episodes with 20,000 validated security overlay attack scenarios targeting seven autonomy layers, including sensing, perception, planning, control, communication, edge/cloud infrastructure, and LLM reasoning. $α^{3}$-SecBench evaluates agents across three orthogonal dimensions: security (attack detection and vulnerability attribution), resilience (safe degradation behavior), and trust (policy-compliant tool usage). We evaluate 23 state-of-the-art LLMs from major industrial providers and leading AI labs using thousands of adversarially augmented UAV episodes sampled from a corpus of 113,475 missions spanning 175 threat types. While many models reliably detect anomalous behavior, effective mitigation, vulnerability attribution, and trustworthy control actions remain inconsistent. Normalized overall scores range from 12.9% to 57.1%, highlighting a significant gap between anomaly detection and security-aware autonomous decision-making. We release $α^{3}$-SecBench on GitHub: https://github.com/maferrag/AlphaSecBench