Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DisasterInsight: A Multimodal Benchmark for Function-Aware and Grounded Disaster Assessment

Sara Tehrani, Yonghao Xu, Leif Haglund, Amanda Berg, Michael Felsberg

TL;DR

Disaster Insight introduces a building-centered multimodal benchmark derived from xBD to enable function-aware and grounded disaster assessment. It combines five tasks—building function, damage level, disaster type classification, counting, and structured reporting—paired with bi-temporal imagery and OpenStreetMap-derived function labels, evaluated under diverse prompts. The authors propose DI-Chat, a domain-adapted, LoRA-finetuned baseline, and show substantial gains over generic VLMs in damage assessment, disaster typing, counting, and report generation, while building-function classification remains challenging. The work demonstrates the importance of task-aligned benchmarks and domain-specific instruction tuning for operational disaster response, providing a pathway toward more trustworthy and grounded multimodal AI in humanitarian contexts.

Abstract

Timely interpretation of satellite imagery is critical for disaster response, yet existing vision-language benchmarks for remote sensing largely focus on coarse labels and image-level recognition, overlooking the functional understanding and instruction robustness required in real humanitarian workflows. We introduce DisasterInsight, a multimodal benchmark designed to evaluate vision-language models (VLMs) on realistic disaster analysis tasks. DisasterInsight restructures the xBD dataset into approximately 112K building-centered instances and supports instruction-diverse evaluation across multiple tasks, including building-function classification, damage-level and disaster-type classification, counting, and structured report generation aligned with humanitarian assessment guidelines. To establish domain-adapted baselines, we propose DI-Chat, obtained by fine-tuning existing VLM backbones on disaster-specific instruction data using parameter-efficient Low-Rank Adaptation (LoRA). Extensive experiments on state-of-the-art generic and remote-sensing VLMs reveal substantial performance gaps across tasks, particularly in damage understanding and structured report generation. DI-Chat achieves significant improvements on damage-level and disaster-type classification as well as report generation quality, while building-function classification remains challenging for all evaluated models. DisasterInsight provides a unified benchmark for studying grounded multimodal reasoning in disaster imagery.

DisasterInsight: A Multimodal Benchmark for Function-Aware and Grounded Disaster Assessment

TL;DR

Disaster Insight introduces a building-centered multimodal benchmark derived from xBD to enable function-aware and grounded disaster assessment. It combines five tasks—building function, damage level, disaster type classification, counting, and structured reporting—paired with bi-temporal imagery and OpenStreetMap-derived function labels, evaluated under diverse prompts. The authors propose DI-Chat, a domain-adapted, LoRA-finetuned baseline, and show substantial gains over generic VLMs in damage assessment, disaster typing, counting, and report generation, while building-function classification remains challenging. The work demonstrates the importance of task-aligned benchmarks and domain-specific instruction tuning for operational disaster response, providing a pathway toward more trustworthy and grounded multimodal AI in humanitarian contexts.

Abstract

Timely interpretation of satellite imagery is critical for disaster response, yet existing vision-language benchmarks for remote sensing largely focus on coarse labels and image-level recognition, overlooking the functional understanding and instruction robustness required in real humanitarian workflows. We introduce DisasterInsight, a multimodal benchmark designed to evaluate vision-language models (VLMs) on realistic disaster analysis tasks. DisasterInsight restructures the xBD dataset into approximately 112K building-centered instances and supports instruction-diverse evaluation across multiple tasks, including building-function classification, damage-level and disaster-type classification, counting, and structured report generation aligned with humanitarian assessment guidelines. To establish domain-adapted baselines, we propose DI-Chat, obtained by fine-tuning existing VLM backbones on disaster-specific instruction data using parameter-efficient Low-Rank Adaptation (LoRA). Extensive experiments on state-of-the-art generic and remote-sensing VLMs reveal substantial performance gaps across tasks, particularly in damage understanding and structured report generation. DI-Chat achieves significant improvements on damage-level and disaster-type classification as well as report generation quality, while building-function classification remains challenging for all evaluated models. DisasterInsight provides a unified benchmark for studying grounded multimodal reasoning in disaster imagery.
Paper Structure (22 sections, 4 figures, 2 tables)

This paper contains 22 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Disaster Assessment Datasets
  4. Vision--Language Models in Remote Sensing
  5. LLM-Based Disaster Reporting
  6. Comparison with Existing Disaster and VLM Datasets
  7. The DisasterInsight Benchmark
  8. Dataset Foundation and Instance Construction
  9. Building Function Annotation from OpenStreetMap
  10. Task Definitions and Granularity
  11. Multi-Prompt Evaluation Strategy
  12. Input Preprocessing and Scene-Level Summarization
  13. Ground-Truth Report Generation and LLM Selection
  14. DI-Chat: Domain-Adapted Instruction-Tuned Baselines
  15. Evaluation Metrics
  16. ...and 7 more sections

Figures (4)

  • Figure 1: Overview of the DisasterInsight benchmark, which defines five tasks spanning object-level recognition and scene-level reasoning on building-centered instances from bi-temporal satellite imagery.
  • Figure 2: Qualitative and dataset-level analysis of DisasterInsight. (a) Word cloud of short-form reports. (b) Word cloud of long-form reports. (c) Training and validation class distributions for BFC, DLC, and DTC; numbers indicate training sample counts.
  • Figure 3: Overview of the DisasterInsight benchmark workflow, illustrating metadata enrichment, offline ground-truth report generation, and multi-task evaluation of vision–language models.
  • Figure 4: Model comparison across tasks and metrics. (a) Report-generation performance. (b) Radar plot summarizing classification, counting, and report generation.