Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

UAVBench and UAVIT-1M: Benchmarking and Enhancing MLLMs for Low-Altitude UAV Vision-Language Understanding

Yang Zhan, Yuan Yuan

Abstract

Multimodal Large Language Models (MLLMs) have made significant strides in natural images and satellite remote sensing images. However, understanding low-altitude drone scenarios remains a challenge. Existing datasets primarily focus on a few specific low-altitude visual tasks, which cannot fully assess the ability of MLLMs in real-world low-altitude UAV applications. Therefore, we introduce UAVBench, a comprehensive benchmark, and UAVIT-1M, a large-scale instruction tuning dataset, designed to evaluate and improve MLLMs' abilities in low-altitude vision-language tasks. UAVBench comprises 43 test units and 966k high-quality data samples across 10 tasks at the image-level and region-level. UAVIT-1M consists of approximately 1.24 million diverse instructions, covering 789k multi-scene images and about 2,000 types of spatial resolutions with 11 distinct tasks. UAVBench and UAVIT-1M feature pure real-world visual images and rich weather conditions, and involve manual verification to ensure high quality. Our in-depth analysis of 11 state-of-the-art MLLMs using UAVBench reveals that open-source MLLMs cannot generate accurate conversations about low-altitude visual content, lagging behind closed-source MLLMs. Extensive experiments demonstrate that fine-tuning open-source MLLMs on UAVIT-1M significantly addresses this gap. Our contributions pave the way for bridging the gap between current MLLMs and low-altitude UAV real-world application demands. (Project page: https://UAVBench.github.io/)

UAVBench and UAVIT-1M: Benchmarking and Enhancing MLLMs for Low-Altitude UAV Vision-Language Understanding

Abstract

Multimodal Large Language Models (MLLMs) have made significant strides in natural images and satellite remote sensing images. However, understanding low-altitude drone scenarios remains a challenge. Existing datasets primarily focus on a few specific low-altitude visual tasks, which cannot fully assess the ability of MLLMs in real-world low-altitude UAV applications. Therefore, we introduce UAVBench, a comprehensive benchmark, and UAVIT-1M, a large-scale instruction tuning dataset, designed to evaluate and improve MLLMs' abilities in low-altitude vision-language tasks. UAVBench comprises 43 test units and 966k high-quality data samples across 10 tasks at the image-level and region-level. UAVIT-1M consists of approximately 1.24 million diverse instructions, covering 789k multi-scene images and about 2,000 types of spatial resolutions with 11 distinct tasks. UAVBench and UAVIT-1M feature pure real-world visual images and rich weather conditions, and involve manual verification to ensure high quality. Our in-depth analysis of 11 state-of-the-art MLLMs using UAVBench reveals that open-source MLLMs cannot generate accurate conversations about low-altitude visual content, lagging behind closed-source MLLMs. Extensive experiments demonstrate that fine-tuning open-source MLLMs on UAVIT-1M significantly addresses this gap. Our contributions pave the way for bridging the gap between current MLLMs and low-altitude UAV real-world application demands. (Project page: https://UAVBench.github.io/)
Paper Structure (30 sections, 3 equations, 24 figures, 14 tables)

This paper contains 30 sections, 3 equations, 24 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Low-Altitude RGB Visual Datasets for UAVs
  4. Low-Altitude Vision-Language Datasets for UAVs
  5. Generic Multimodal Large Language Models
  6. Remote Sensing Multimodal Large Language Models
  7. UAVBench Benchmark and UAVIT-1M Dataset
  8. Data Processing and Benchmark Construction
  9. Step 1: Data preparation
  10. Step 2: Data Integration
  11. Step 3: Data Generation
  12. Step 4: Quality Control
  13. Instruction-Tuning Dataset Construction
  14. Dataset Statistics
  15. Multimodal Fusion and Alignment Methodology
  16. ...and 15 more sections

Figures (24)

  • Figure 1: Sampled UAVIT-1M examples from our designed 11 tasks. We present the images, the corresponding questions, and the answers. These tasks are divided into two levels, namely, Image-level: Image Classification, Detailed Classification, Target Counting, Image Captioning, Detailed Image Captioning, and Image VQA; Region-level: Region VQA, Region Captioning, Region Classification, Region Detection, and Visual Grounding.
  • Figure 2: An overview of (a) data processing for UAVBench and (b) instruction construction pipeline for UAVIT-1M.
  • Figure 3: Category distribution in each task. Zoom in to view the specific categories and corresponding quantities.
  • Figure 4: Image resolution and target position distributions. Best viewed by zooming in.
  • Figure 5: Some typical statistics of UAVBench and UAVIT-1M. (a) Distribution of question types in image and region-level VQA tasks. (b) Distribution of object sizes in all region-level tasks. (c) Distribution of difficulty in the target counting task.
  • ...and 19 more figures