Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada

Braeden Sherritt, Isar Nejadgholi, Efstratios Aivaliotis, Khaled Mslmani, Marzieh Amini

TL;DR

This work addresses the need for Canada-specific, real-time wildfire information from social media by introducing WildFireCan-MMD, a multimodal dataset of 4,688 image-text posts across Canadian wildfires (2022–2024) annotated with a 12-category taxonomy derived via BERTopic. It benchmarks multiple approaches, finding that a task-specific, fine-tuned dual-encoder transformer model (ViT for images and RoBERTa for text with a transformer fusion module) achieves the best overall F1 score of 84.48%, outperforming zero-shot vision-language models. The study also demonstrates the value of leveraging unlabeled data to uncover trends in wildfire-related discourse from 2018–2024. Overall, the dataset and modeling work provide a targeted resource for improving disaster response analytics and highlight the importance of localized, domain-specific datasets and training.

Abstract

Rapid information access is vital during wildfires, yet traditional data sources are slow and costly. Social media offers real-time updates, but extracting relevant insights remains a challenge. In this work, we focus on multimodal wildfire social media data, which, although existing in current datasets, is currently underrepresented in Canadian contexts. We present WildFireCan-MMD, a new multimodal dataset of X posts from recent Canadian wildfires, annotated across twelve key themes. We evaluate zero-shot vision-language models on this dataset and compare their results with those of custom-trained and baseline classifiers. We show that while baseline methods and zero-shot prompting offer quick deployment, custom-trained models outperform them when labelled data is available. Our best-performing custom model reaches 84.48% f-score, outperforming VLMs and baseline classifiers. We also demonstrate how this model can be used to uncover trends during wildfires, through the collection and analysis of a large unlabeled dataset. Our dataset facilitates future research in wildfire response, and our findings highlight the importance of tailored datasets and task-specific training. Importantly, such datasets should be localized, as disaster response requirements vary across regions and contexts.

WildFireCan-MMD: A Multimodal Dataset for Classification of User-Generated Content During Wildfires in Canada

TL;DR

This work addresses the need for Canada-specific, real-time wildfire information from social media by introducing WildFireCan-MMD, a multimodal dataset of 4,688 image-text posts across Canadian wildfires (2022–2024) annotated with a 12-category taxonomy derived via BERTopic. It benchmarks multiple approaches, finding that a task-specific, fine-tuned dual-encoder transformer model (ViT for images and RoBERTa for text with a transformer fusion module) achieves the best overall F1 score of 84.48%, outperforming zero-shot vision-language models. The study also demonstrates the value of leveraging unlabeled data to uncover trends in wildfire-related discourse from 2018–2024. Overall, the dataset and modeling work provide a targeted resource for improving disaster response analytics and highlight the importance of localized, domain-specific datasets and training.

Abstract

Rapid information access is vital during wildfires, yet traditional data sources are slow and costly. Social media offers real-time updates, but extracting relevant insights remains a challenge. In this work, we focus on multimodal wildfire social media data, which, although existing in current datasets, is currently underrepresented in Canadian contexts. We present WildFireCan-MMD, a new multimodal dataset of X posts from recent Canadian wildfires, annotated across twelve key themes. We evaluate zero-shot vision-language models on this dataset and compare their results with those of custom-trained and baseline classifiers. We show that while baseline methods and zero-shot prompting offer quick deployment, custom-trained models outperform them when labelled data is available. Our best-performing custom model reaches 84.48% f-score, outperforming VLMs and baseline classifiers. We also demonstrate how this model can be used to uncover trends during wildfires, through the collection and analysis of a large unlabeled dataset. Our dataset facilitates future research in wildfire response, and our findings highlight the importance of tailored datasets and task-specific training. Importantly, such datasets should be localized, as disaster response requirements vary across regions and contexts.

Paper Structure

This paper contains 24 sections, 9 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Background and Literature Review
  3. Social media analysis for disaster response
  4. Multimodal classification for disaster response
  5. The WildFireCan-MMD Dataset
  6. Data Collection
  7. Taxonomy Development
  8. Data Labeling
  9. Classification Methodology
  10. Experimental Results
  11. Discussion
  12. Analysis on Trends in Data
  13. Conclusion
  14. Annotation
  15. Topic Modeling
  16. ...and 9 more sections

Figures (9)

  • Figure 1: Examples of labeled posts from WildFireCan-MMD.
  • Figure 2: Distribution of collected tweets based on author location.
  • Figure 3: Weekly post distributions: (left) overall yearly trend, (right) class-wise distribution in 2023.
  • Figure A.1: Counts of labels by each annotator
  • Figure B.1: Examples of useful/non-useful topics from topic modeling on our dataset.
  • ...and 4 more figures