Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features

Muhammad Imran, Abdul Wahab Ziaullah, Kai Chen, Ferda Ofli

TL;DR

This study methodically assesses the robustness of six LLMs on crisis-related microblogs across 19 disasters using the HumAID dataset. It compares proprietary GPT-3.5/4/4o and open-source Llama/Mistral models under zero-shot and few-shot settings, across disaster types, information categories, native vs non-native English data, and linguistic features. Key findings show GPT-4o and GPT-4 offer stronger cross-event generalization but struggle with flood data and urgent-need classifications, with few-shot cues providing limited gains; open-source models generally lag, though some exceptions occur. The work highlights specific weaknesses and suggests directions for developing more robust, linguistically aware crisis-response systems and potential multimodal extensions for comprehensive disaster management.

Abstract

The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.

Evaluating Robustness of LLMs on Crisis-Related Microblogs across Events, Information Types, and Linguistic Features

TL;DR

This study methodically assesses the robustness of six LLMs on crisis-related microblogs across 19 disasters using the HumAID dataset. It compares proprietary GPT-3.5/4/4o and open-source Llama/Mistral models under zero-shot and few-shot settings, across disaster types, information categories, native vs non-native English data, and linguistic features. Key findings show GPT-4o and GPT-4 offer stronger cross-event generalization but struggle with flood data and urgent-need classifications, with few-shot cues providing limited gains; open-source models generally lag, though some exceptions occur. The work highlights specific weaknesses and suggests directions for developing more robust, linguistically aware crisis-response systems and potential multimodal extensions for comprehensive disaster management.

Abstract

The widespread use of microblogging platforms like X (formerly Twitter) during disasters provides real-time information to governments and response authorities. However, the data from these platforms is often noisy, requiring automated methods to filter relevant information. Traditionally, supervised machine learning models have been used, but they lack generalizability. In contrast, Large Language Models (LLMs) show better capabilities in understanding and processing natural language out of the box. This paper provides a detailed analysis of the performance of six well-known LLMs in processing disaster-related social media data from a large-set of real-world events. Our findings indicate that while LLMs, particularly GPT-4o and GPT-4, offer better generalizability across different disasters and information types, most LLMs face challenges in processing flood-related data, show minimal improvement despite the provision of examples (i.e., shots), and struggle to identify critical information categories like urgent requests and needs. Additionally, we examine how various linguistic features affect model performance and highlight LLMs' vulnerabilities against certain features like typos. Lastly, we provide benchmarking results for all events across both zero- and few-shot settings and observe that proprietary models outperform open-source ones in all tasks.

Paper Structure

This paper contains 16 sections, 11 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Assessment Methodology
  4. Dataset and Models
  5. Models
  6. Experimental Design
  7. Results and Discussion
  8. Disaster Type Analysis
  9. Information Type Analysis
  10. Native vs. Non-Native English Analysis
  11. Linguistic Feature Analysis
  12. Hashtag Positioning Impacts
  13. Event-wise and Overall Performance
  14. Ethical Considerations
  15. Conclusion and Future Work
  16. ...and 1 more sections

Figures (11)

  • Figure 1: Data distributions for (a) events, (b) information types/classes, (c) disaster types, and (d) native/non-native English countries
  • Figure 2: Performance (F1-scores) of LLMs across disaster types and few-shot settings
  • Figure 3: Performance (F1 scores) of LLMs across various information types (i.e., classes)
  • Figure 4: Confusion matrices for GPT-4o (left) and Mistral 7B (right) models under the zero-shot setting
  • Figure 5: Performance (F1-scores) of LLMs on native-English-speaking vs. non-English-speaking countries. LM2=Llama-2 13B, LM3=Llama-3 8B, MST=Mistral 7B
  • ...and 6 more figures