DisastQA: A Comprehensive Benchmark for Evaluating Question Answering in Disaster Management

TL;DR

DisastQA confronts the critical need for reliable question answering in disaster management by introducing a large-scale, human-verified benchmark that combines multiple-choice and open-ended tasks across eight disaster types. The authors employ a Human–LLM collaboration pipeline to generate high-quality questions grounded in real-world evidence and introduce a keypoint-based evaluation for open-ended responses, enabling rigorous measurement of factual completeness under controlled evidence settings (Base, Mix, Golden). Evaluations across 20 models reveal that while open-weight models close some gaps to proprietary systems in clean contexts, performance deteriorates substantially under noisy evidence, highlighting persistent reliability gaps for disaster response. The work provides a transferable benchmark, a principled evaluation protocol, and extensive results that emphasize the need for domain-specific reliability beyond general-domain fluency, with resources released for future research.

Abstract

Accurate question answering (QA) in disaster management requires reasoning over uncertain and conflicting information, a setting poorly captured by existing benchmarks built on clean evidence. We introduce DisastQA, a large-scale benchmark of 3,000 rigorously verified questions (2,000 multiple-choice and 1,000 open-ended) spanning eight disaster types. The benchmark is constructed via a human-LLM collaboration pipeline with stratified sampling to ensure balanced coverage. Models are evaluated under varying evidence conditions, from closed-book to noisy evidence integration, enabling separation of internal knowledge from reasoning under imperfect information. For open-ended QA, we propose a human-verified keypoint-based evaluation protocol emphasizing factual completeness over verbosity. Experiments with 20 models reveal substantial divergences from general-purpose leaderboards such as MMLU-Pro. While recent open-weight models approach proprietary systems in clean settings, performance degrades sharply under realistic noise, exposing critical reliability gaps for disaster response. All code, data, and evaluation resources are available at https://github.com/TamuChen18/DisastQA_open.

Figures (5)

• Figure 1: Model ranking divergence between DisastQA-MCQ (Gold) and the MMLU-Pro subset. Pronounced off-diagonal deviations indicate that general QA leaderboards may not reliably predict relative performance in high-stakes disaster-response QA. Full numerical results are provided in Table \ref{['tab:mmlu_subset']}.
• Figure 2: The Human--LLM Collaborative Pipeline for DisastQA. User queries are stratified across 8 disaster event types (Top). Two parallel tracks construct the dataset: the MCQ Pipeline with human refinement for distractor quality and answer correctness, and the OE Pipeline with human rewriting and Keypoint annotation (Middle). Models are evaluated under three evidence settings (Base, Mix, Golden) (Bottom); see Section \ref{['sec:evaluation_methodology']} for details.
• Figure 3: (a) Distribution of keypoint counts per OE answer. (b) Relationship between answer length (in tokens) and keypoint count, showing that longer answers typically contain more keypoints.
• Figure 4: Keypoint Coverage across difficulty levels under Base, Mix, and Golden.
• Figure 5: MCQ Accuracy breakdown by Event Type. The gap between Base and Golden is most pronounced in specialized domains (e.g., Biological, Extraterrestrial), confirming that models rely on retrieval for long-tail knowledge.