Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark

Haodong Li, Haicheng Qu, Xiaofeng Zhang

TL;DR

A high quality remote sensing LVLMs dataset, DDFAV, is introduced using data augmentation and data mixing strategies and a hallucination evaluation method RSPOPE is developed based on the proposed dataset and evaluated to evaluate the zero-shot capabilities of different LVLMs.

Abstract

With the rapid development of large vision language models (LVLMs), these models have shown excellent results in various multimodal tasks. Since LVLMs are prone to hallucinations and there are currently few datasets and evaluation methods specifically designed for remote sensing, their performance is typically poor when applied to remote sensing tasks. To address these issues, this paper introduces a high quality remote sensing LVLMs dataset, DDFAV, created using data augmentation and data mixing strategies. Next, a training instruction set is produced based on some high-quality remote sensing images selected from the proposed dataset. Finally, we develop a remote sensing LVLMs hallucination evaluation method RSPOPE based on the proposed dataset and evaluate the zero-shot capabilities of different LVLMs. Our proposed dataset, instruction set, and evaluation method files are available at https://github.com/HaodongLi2024/rspope.

DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark

TL;DR

A high quality remote sensing LVLMs dataset, DDFAV, is introduced using data augmentation and data mixing strategies and a hallucination evaluation method RSPOPE is developed based on the proposed dataset and evaluated to evaluate the zero-shot capabilities of different LVLMs.

Abstract

With the rapid development of large vision language models (LVLMs), these models have shown excellent results in various multimodal tasks. Since LVLMs are prone to hallucinations and there are currently few datasets and evaluation methods specifically designed for remote sensing, their performance is typically poor when applied to remote sensing tasks. To address these issues, this paper introduces a high quality remote sensing LVLMs dataset, DDFAV, created using data augmentation and data mixing strategies. Next, a training instruction set is produced based on some high-quality remote sensing images selected from the proposed dataset. Finally, we develop a remote sensing LVLMs hallucination evaluation method RSPOPE based on the proposed dataset and evaluate the zero-shot capabilities of different LVLMs. Our proposed dataset, instruction set, and evaluation method files are available at https://github.com/HaodongLi2024/rspope.

Paper Structure

This paper contains 8 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. DDFAV Remote Sensing LVLMs Dataset
  4. DDFAV Remote Sensing LVLMs Instruction Set
  5. Experiments and Analysis
  6. POPE Metric Evaluation of Remote Sensing Datasets
  7. Comparison of Zero-Shot Capabilities of Different General LVLMs for Remote Sensing Data
  8. Conclusion

Figures (4)

  • Figure 1: Comparison of hallucination phenomena of general LVLMs in image captioning tasks for remote sensing images.
  • Figure 2: The number of object categories and source distribution of our proposed remote sensing LVLMs dataset DDFAV.
  • Figure 3: An example from the DDFAV remote sensing LVLMs instruction set includes a remote sensing image with 8 question-answer pairs: 1 detailed image description, 1 complex reasoning question, 2 visual questions about color, 2 visual questions about counting, and 2 visual questions about object location.
  • Figure 4: The RSPOPE evaluation benchmark is designed for specific images in the DDFAV dataset. From top to bottom, the settings of the three pictures are easy + random, medium + popular, and hard + adversarial.