Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

Aisha Al-Mohannadi, Ayisha Firoz, Yin Yang, Muhammad Imran, Ferda Ofli

TL;DR

DisasterVQA introduces a ground-level disaster visual question answering benchmark designed to stress perception and reasoning under crisis conditions. The dataset combines 1,395 real-world images with 4,405 questions across binary, multiple-choice, and open-ended formats, all grounded in humanitarian frameworks like FEMA ESF and OCHA MIRA. A multi-stage pipeline—image selection, GPT-4o question generation, human verification, and LLM-based post-processing—yields rich QA pairs evaluated against seven state-of-the-art vision–language models, with results showing strong binary performance but notable gaps in fine-grained reasoning, counting, and region-specific understanding. The work provides a practical evaluation framework and baseline insights to drive the development of more reliable, operationally meaningful VLMs for disaster response, and it makes the dataset publicly available for reproducibility and future research.

Abstract

Social media imagery provides a low-latency source of situational information during natural and human-induced disasters, enabling rapid damage assessment and response. While Visual Question Answering (VQA) has shown strong performance in general-purpose domains, its suitability for the complex and safety-critical reasoning required in disaster response remains unclear. We introduce DisasterVQA, a benchmark dataset designed for perception and reasoning in crisis contexts. DisasterVQA consists of 1,395 real-world images and 4,405 expert-curated question-answer pairs spanning diverse events such as floods, wildfires, and earthquakes. Grounded in humanitarian frameworks including FEMA ESF and OCHA MIRA, the dataset includes binary, multiple-choice, and open-ended questions covering situational awareness and operational decision-making tasks. We benchmark seven state-of-the-art vision-language models and find performance variability across question types, disaster categories, regions, and humanitarian tasks. Although models achieve high accuracy on binary questions, they struggle with fine-grained quantitative reasoning, object counting, and context-sensitive interpretation, particularly for underrepresented disaster scenarios. DisasterVQA provides a challenging and practical benchmark to guide the development of more robust and operationally meaningful vision-language models for disaster response. The dataset is publicly available at https://zenodo.org/records/18267770.

DisasterVQA: A Visual Question Answering Benchmark Dataset for Disaster Scenes

TL;DR

DisasterVQA introduces a ground-level disaster visual question answering benchmark designed to stress perception and reasoning under crisis conditions. The dataset combines 1,395 real-world images with 4,405 questions across binary, multiple-choice, and open-ended formats, all grounded in humanitarian frameworks like FEMA ESF and OCHA MIRA. A multi-stage pipeline—image selection, GPT-4o question generation, human verification, and LLM-based post-processing—yields rich QA pairs evaluated against seven state-of-the-art vision–language models, with results showing strong binary performance but notable gaps in fine-grained reasoning, counting, and region-specific understanding. The work provides a practical evaluation framework and baseline insights to drive the development of more reliable, operationally meaningful VLMs for disaster response, and it makes the dataset publicly available for reproducibility and future research.

Abstract

Social media imagery provides a low-latency source of situational information during natural and human-induced disasters, enabling rapid damage assessment and response. While Visual Question Answering (VQA) has shown strong performance in general-purpose domains, its suitability for the complex and safety-critical reasoning required in disaster response remains unclear. We introduce DisasterVQA, a benchmark dataset designed for perception and reasoning in crisis contexts. DisasterVQA consists of 1,395 real-world images and 4,405 expert-curated question-answer pairs spanning diverse events such as floods, wildfires, and earthquakes. Grounded in humanitarian frameworks including FEMA ESF and OCHA MIRA, the dataset includes binary, multiple-choice, and open-ended questions covering situational awareness and operational decision-making tasks. We benchmark seven state-of-the-art vision-language models and find performance variability across question types, disaster categories, regions, and humanitarian tasks. Although models achieve high accuracy on binary questions, they struggle with fine-grained quantitative reasoning, object counting, and context-sensitive interpretation, particularly for underrepresented disaster scenarios. DisasterVQA provides a challenging and practical benchmark to guide the development of more robust and operationally meaningful vision-language models for disaster response. The dataset is publicly available at https://zenodo.org/records/18267770.
Paper Structure (32 sections, 8 figures)

This paper contains 32 sections, 8 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. DisasterVQA Construction
  4. Image Selection
  5. Question Generation
  6. Question Verification
  7. Humanitarian Mapping
  8. Final Dataset Statistics
  9. Benchmarking Experiments
  10. Vision-Language Models
  11. Output Post-processing
  12. Evaluation Metrics
  13. Computational Resources
  14. Experimental Results
  15. Analysis By Humanitarian Category
  16. ...and 17 more sections

Figures (8)

  • Figure 1: Sample image-question-answer triplets from the proposed DisasterVQA dataset
  • Figure 2: Overview of the four stages followed for the dataset curation and benchmarking
  • Figure 3: Overview of dataset composition in terms of image and question counts across (a) disaster types, (b) geographic regions, and (c) humanitarian categories.
  • Figure 4: Overall performance of all the models for binary, open-ended, and multiple-choice questions
  • Figure 5: Performance by humanitarian categories across models. ST: Situational Awareness, AT: Actionable
  • ...and 3 more figures