MM-UAVBench: How Well Do Multimodal Large Language Models See, Think, and Plan in Low-Altitude UAV Scenarios?

TL;DR

MM-UAVBench delivers a real-world, multi-task benchmark to probe how well multimodal large language models can see, think, and plan in low-altitude UAV contexts. By assembling 19 tasks across perception, cognition, and planning from diverse UAV data, the authors reveal that current MLLMs struggle with UAV-specific challenges such as object-scale variation, spatial biases, and multi-view fusion, despite improvements with larger models and CoT prompting. The work provides a rigorous evaluation framework, datasets, and analyses (including egocentric planning and multi-view reasoning) that pinpoint bottlenecks and guide UAV-tailored MLLM design. The benchmark’s findings have practical implications for deploying reliable autonomous aerial intelligence in real-world missions and spurring targeted research in UAV-oriented multimodal reasoning.

Abstract

While Multimodal Large Language Models (MLLMs) have exhibited remarkable general intelligence across diverse domains, their potential in low-altitude applications dominated by Unmanned Aerial Vehicles (UAVs) remains largely underexplored. Existing MLLM benchmarks rarely cover the unique challenges of low-altitude scenarios, while UAV-related evaluations mainly focus on specific tasks such as localization or navigation, without a unified evaluation of MLLMs'general intelligence. To bridge this gap, we present MM-UAVBench, a comprehensive benchmark that systematically evaluates MLLMs across three core capability dimensions-perception, cognition, and planning-in low-altitude UAV scenarios. MM-UAVBench comprises 19 sub-tasks with over 5.7K manually annotated questions, all derived from real-world UAV data collected from public datasets. Extensive experiments on 16 open-source and proprietary MLLMs reveal that current models struggle to adapt to the complex visual and cognitive demands of low-altitude scenarios. Our analyses further uncover critical bottlenecks such as spatial bias and multi-view understanding that hinder the effective deployment of MLLMs in UAV scenarios. We hope MM-UAVBench will foster future research on robust and reliable MLLMs for real-world UAV intelligence.

Figures (16)

• Figure 1: Overview of MM-UAVBench. MM-UAVBench consists of 19 tasks covering three core capability dimensions: Perception, Cognition, and Planning. Perception tasks assess basic visual understanding such as classification, OCR, and counting. Cognition tasks span three hierarchical levels—object-level, scene-level, and event-level—evaluating the model’s ability to infer intentions, reason across objects, analyze scenes, understand events, and predict outcomes. Planning tasks assess UAV-specific decision making, including planning for single or multi-UAV systems, directing ground-target actions from an aerial perspective, and coordinating cooperative actions between aerial agents and ground participants. All examples shown are real UAV imagery, illustrating the diverse challenges present in low-altitude scenarios.
• Figure 2: The task design of MM-UAVBench covers 3 high-level categories, 8 sub-catigories and 19 fine-grained tasks in MM-UAVBench.
• Figure 3: Statistics of MM-UAVBench.(a) Distribution of the 19 sub-tasks. (b) Proportions of the three input modalities. (c) Annotation metrics, where $N_{\text{video}}$ and $N_{\text{img}}$ denote the numbers of video clips and images; $\text{Avg.\ Res.}$ denotes the average resolution; $N_{\mathrm{bbox,\,man}}$, $N_{\mathrm{bbox,\,obj}}$, and $N_{\mathrm{bbox,\,reg}}$ denote the numbers of human, object, and region bounding boxes; $\text{Avg.\ }S_{\text{man}}$, $\text{Avg.\ }S_{\text{obj}}$, and $\text{Avg.\ }S_{\text{reg}}$ represent their average area ratios.
• Figure 4: Accuracy comparison across small, medium, and large target sizes on Orient. Classification and Target Backtracking tasks.
• Figure 5: Confusion matrices of predicted (P.) directions from MLLMs versus ground-truth (T.) on the Orient. Classification task.