Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Detecting Actionable Requests and Offers on Social Media During Crises Using LLMs

Ahmed El Fekih Zguir, Ferda Ofli, Muhammad Imran

TL;DR

The paper tackles the challenge of identifying actionable requests and offers in social media during disasters by introducing a fine-grained top-down taxonomy across supplies, actions, and emergency personnel. It couples this taxonomy with a Query-Specific Few-shot Learning (QSF Learning) framework that uses Retrieval-Augmented Generation to assemble query-relevant labeled examples for each input, enabling robust multi-label classification across multiple LLMs. The authors evaluate on both synthetic data produced with GPT-4o and real-world Hurricane Sandy data, demonstrating that QSF Learning consistently outperforms baseline prompting strategies, improves actionability prioritization, and narrows performance gaps between model sizes. Findings highlight the importance of granular taxonomy design and dynamic few-shot prompts for crisis response, with practical implications for accelerating humanitarian actions and resource allocation in real-time during crises.

Abstract

Natural disasters often result in a surge of social media activity, including requests for assistance, offers of help, sentiments, and general updates. To enable humanitarian organizations to respond more efficiently, we propose a fine-grained hierarchical taxonomy to systematically organize crisis-related information about requests and offers into three critical dimensions: supplies, emergency personnel, and actions. Leveraging the capabilities of Large Language Models (LLMs), we introduce Query-Specific Few-shot Learning (QSF Learning) that retrieves class-specific labeled examples from an embedding database to enhance the model's performance in detecting and classifying posts. Beyond classification, we assess the actionability of messages to prioritize posts requiring immediate attention. Extensive experiments demonstrate that our approach outperforms baseline prompting strategies, effectively identifying and prioritizing actionable requests and offers.

Detecting Actionable Requests and Offers on Social Media During Crises Using LLMs

TL;DR

The paper tackles the challenge of identifying actionable requests and offers in social media during disasters by introducing a fine-grained top-down taxonomy across supplies, actions, and emergency personnel. It couples this taxonomy with a Query-Specific Few-shot Learning (QSF Learning) framework that uses Retrieval-Augmented Generation to assemble query-relevant labeled examples for each input, enabling robust multi-label classification across multiple LLMs. The authors evaluate on both synthetic data produced with GPT-4o and real-world Hurricane Sandy data, demonstrating that QSF Learning consistently outperforms baseline prompting strategies, improves actionability prioritization, and narrows performance gaps between model sizes. Findings highlight the importance of granular taxonomy design and dynamic few-shot prompts for crisis response, with practical implications for accelerating humanitarian actions and resource allocation in real-time during crises.

Abstract

Natural disasters often result in a surge of social media activity, including requests for assistance, offers of help, sentiments, and general updates. To enable humanitarian organizations to respond more efficiently, we propose a fine-grained hierarchical taxonomy to systematically organize crisis-related information about requests and offers into three critical dimensions: supplies, emergency personnel, and actions. Leveraging the capabilities of Large Language Models (LLMs), we introduce Query-Specific Few-shot Learning (QSF Learning) that retrieves class-specific labeled examples from an embedding database to enhance the model's performance in detecting and classifying posts. Beyond classification, we assess the actionability of messages to prioritize posts requiring immediate attention. Extensive experiments demonstrate that our approach outperforms baseline prompting strategies, effectively identifying and prioritizing actionable requests and offers.

Paper Structure

This paper contains 16 sections, 1 equation, 6 figures, 4 tables.

Table of Contents

  1. Keywords
  2. Introduction
  3. Related Work
  4. Methodology
  5. Taxonomy Generation
  6. Task Definition and Classifiers
  7. Task Definition
  8. Baseline classifiers
  9. Query-Specific Few-Shot Learning Approach
  10. Data and Evaluation
  11. Synthetic Data Generation
  12. Real-World Data
  13. Evaluation metrics
  14. Results and Discussion
  15. Limitations and Future work
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Request and Offer Taxonomy: A partial representation
  • Figure 2: Diagram of baseline (BL) classifiers. Each BL has a different prompt in the diagram.
  • Figure 3: A high-level overview of the Query-Specific Few-shot Learning approach
  • Figure 4: Distribution of the synthetic generated data across (a) Type and (b) Supply categories
  • Figure 5: Distribution of the synthetic generated data across (a) Action and (b) Emergency Personnel categories
  • ...and 1 more figures