CrisisViT: A Robust Vision Transformer for Crisis Image Classification

TL;DR

This work tackles the need for rapid, accurate crisis-image classification from abundant citizen-posted imagery. It introduces CrisisViT, a Vision Transformer pretrained on the in-domain Incidents1M crisis image dataset, and evaluates it on the Crisis Image Benchmark across four tasks. Findings show that CrisisViT generally outperforms CNN baselines, with in-domain pretraining delivering 0.9–1.5% absolute gains and an overall average improvement of 1.25% across tasks; placing labels in Incidents1M yield the strongest benefits, while combining with ImageNet-1k offers mixed results. The study also provides an in-depth analysis of pretraining strategies and releases the CrisisViT models to support real-world crisis-response applications.

Abstract

In times of emergency, crisis response agencies need to quickly and accurately assess the situation on the ground in order to deploy relevant services and resources. However, authorities often have to make decisions based on limited information, as data on affected regions can be scarce until local response services can provide first-hand reports. Fortunately, the widespread availability of smartphones with high-quality cameras has made citizen journalism through social media a valuable source of information for crisis responders. However, analyzing the large volume of images posted by citizens requires more time and effort than is typically available. To address this issue, this paper proposes the use of state-of-the-art deep neural models for automatic image classification/tagging, specifically by adapting transformer-based architectures for crisis image classification (CrisisViT). We leverage the new Incidents1M crisis image dataset to develop a range of new transformer-based image classification models. Through experimentation over the standard Crisis image benchmark dataset, we demonstrate that the CrisisViT models significantly outperform previous approaches in emergency type, image relevance, humanitarian category, and damage severity classification. Additionally, we show that the new Incidents1M dataset can further augment the CrisisViT models resulting in an additional 1.25% absolute accuracy gain.