Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation

Aissatou Diallo, Antonis Bikakis, Luke Dickens, Anthony Hunter, Rob Miller

TL;DR

This work introduces RESPONSE, a large, human-curated dataset designed to benchmark how language models perform commonsense reasoning in disaster scenarios, including time-sensitive decisions and explanations validated by domain experts. By evaluating GPT-4 and Claude-3 on both short and longer horizons, the study reveals that even top models achieve only a fraction of human-level performance for immediate actions and that traditional automatic metrics poorly reflect true usefulness or safety. The authors advocate for more nuanced evaluation frameworks that capture practical feasibility and alignment with human judgment in crises, and they release the dataset and tooling to spur further development. Overall, RESPONSE highlights substantial gaps in current LLMs' crisis reasoning and offers a path toward more reliable decision-support tools in emergency management.

Abstract

An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present \textsf{RESPONSE}, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37\% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.

RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation

TL;DR

This work introduces RESPONSE, a large, human-curated dataset designed to benchmark how language models perform commonsense reasoning in disaster scenarios, including time-sensitive decisions and explanations validated by domain experts. By evaluating GPT-4 and Claude-3 on both short and longer horizons, the study reveals that even top models achieve only a fraction of human-level performance for immediate actions and that traditional automatic metrics poorly reflect true usefulness or safety. The authors advocate for more nuanced evaluation frameworks that capture practical feasibility and alignment with human judgment in crises, and they release the dataset and tooling to spur further development. Overall, RESPONSE highlights substantial gaps in current LLMs' crisis reasoning and offers a path toward more reliable decision-support tools in emergency management.

Abstract

An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present \textsf{RESPONSE}, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs' commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37\% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs' ability for commonsense reasoning in crises.

Paper Structure

This paper contains 41 sections, 6 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Skill evaluation for LLMs
  4. LLMs and Human Behaviour
  5. LLMs for Disaster Management
  6. The RESPONSE Dataset
  7. Motivation
  8. Closer to a probe than a benchmark
  9. Task Formulation
  10. Preprocessing
  11. Data Collection
  12. Natural Language Question Generation
  13. Entry Contents
  14. Experiments
  15. Experimental Setup
  16. ...and 26 more sections

Figures (6)

  • Figure 1: A sample from RESPONSE for the "immediate" time frame. The type of incident and location are (wildfire, highway).
  • Figure 2: Data-generating process for the RESPONSE dataset. Top: Image and labels annotated with problem, missing resource (MR), solutions, and explanations. Bottom: Natural language question generation based on problem, missing resource, and time frame.
  • Figure 3: Pie chart representing classes for the missing resources in RESPONSE.
  • Figure 4: Histograms of location in RESPONSE
  • Figure 5: Histograms of incidents in RESPONSE
  • ...and 1 more figures