Design and evaluation of an agentic workflow for crisis-related synthetic tweet datasets

Abstract

Twitter (now X) has become an important source of social media data for situational awareness during crises. Crisis informatics research has widely used tweets from Twitter to develop and evaluate artificial intelligence (AI) systems for various crisis-relevant tasks, such as extracting locations and estimating damage levels from tweets to support damage assessment. However, recent changes in Twitter's data access policies have made it increasingly difficult to curate real-world tweet datasets related to crises. Moreover, existing curated tweet datasets are limited to past crisis events in specific contexts and are costly to annotate at scale. These limitations constrain the development and evaluation of AI systems used in crisis informatics. To address these limitations, we introduce an agentic workflow for generating crisis-related synthetic tweet datasets. The workflow iteratively generates synthetic tweets conditioned on prespecified target characteristics, evaluates them using predefined compliance checks, and incorporates structured feedback to refine them in subsequent iterations. As a case study, we apply the workflow to generate synthetic tweet datasets relevant to post-earthquake damage assessment. We show that the workflow can generate synthetic tweets that capture their target labels for location and damage level. We further demonstrate that the resulting synthetic tweet datasets can be used to evaluate AI systems on damage assessment tasks like geolocalization and damage level prediction. Our results indicate that the workflow offers a flexible and scalable alternative to real-world tweet data curation, enabling the systematic generation of synthetic social media data across diverse crisis events, societal contexts, and crisis informatics applications.

Figures (7)

• Figure 1: Schematic overview of the agentic workflow for generating crisis-related synthetic tweet datasets. The workflow includes three agents: (1) a tweet generator $g$ that generates synthetic tweets, (2) a compliance evaluator $e$ that evaluates those tweets, and (3) a feedback augmenter $a$ that provides feedback to the tweet generator $g$ based on the evaluations of the compliance evaluator $e$. The workflow iterates for $n$ rounds, with synthetic tweets that gets accepted by the compliance evaluator $e$ at each round added to the synthetic tweet dataset $\mathcal{D}_{syn}$.
• Figure 2: Percentage of synthetic tweets that passed compliance checks for location correctness, damage level correctness, and textual diversity. In the initial generation attempt, at least 30% of synthetic tweets passed each compliance check. This percentage improved with every additional feedback round.
• Figure 3: Distribution of synthetic tweets by the number of compliance checks passed across the six earthquake events. The percentage of accepted synthetic tweets increases with more feedback rounds.
• Figure 4: Percentage of tweets that passed the compliance checks when using different temperature $\tau$ values. Higher temperature values generated more diverse synthetic tweets, resulting in more synthetic tweets that passed the textual diversity check and were eventually accepted and added to the synthetic tweet dataset. Results shown are based on the 2014 Napa, California earthquake event.
• Figure 5: Tweet length distributions and relative hashtag frequencies in the real and synthetic tweet datasets. The synthetic tweets reflect common characteristics of real tweets, such as character limits and the use of common hashtags during crises (e.g., #earthquake, #napa, #iquique, #nepal, #ridgecrest, #fukushima, #haiti).