Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media

Thi Huyen Nguyen, Koustav Rudra, Wolfgang Nejdl

Abstract

Advances in social media data dissemination enable the provision of real-time information during a crisis. The information comes from different classes, such as infrastructure damages, persons missing or stranded in the affected zone, etc. Existing methods attempted to classify text and images into various humanitarian categories, but their decision-making process remains largely opaque, which affects their deployment in real-life applications. Recent work has sought to improve transparency by extracting textual rationales from tweets to explain predicted classes. However, such explainable classification methods have mostly focused on text, rather than crisis-related images. In this paper, we propose an interpretable-by-design multimodal classification framework. Our method first learns the joint representation of text and image using a visual language transformer model and extracts text rationales. Next, it extracts the image rationales via the mapping with text rationales. Our approach demonstrates how to learn rationales in one modality from another through cross-modal rationale transfer, which saves annotation effort. Finally, tweets are classified based on extracted rationales. Experiments are conducted over CrisisMMD benchmark dataset, and results show that our proposed method boosts the classification Macro-F1 by 2-35% while extracting accurate text tokens and image patches as rationales. Human evaluation also supports the claim that our proposed method is able to retrieve better image rationale patches (12%) that help to identify humanitarian classes. Our method adapts well to new, unseen datasets in zero-shot mode, achieving an accuracy of 80%.

Cross-Modal Rationale Transfer for Explainable Humanitarian Classification on Social Media

Abstract

Advances in social media data dissemination enable the provision of real-time information during a crisis. The information comes from different classes, such as infrastructure damages, persons missing or stranded in the affected zone, etc. Existing methods attempted to classify text and images into various humanitarian categories, but their decision-making process remains largely opaque, which affects their deployment in real-life applications. Recent work has sought to improve transparency by extracting textual rationales from tweets to explain predicted classes. However, such explainable classification methods have mostly focused on text, rather than crisis-related images. In this paper, we propose an interpretable-by-design multimodal classification framework. Our method first learns the joint representation of text and image using a visual language transformer model and extracts text rationales. Next, it extracts the image rationales via the mapping with text rationales. Our approach demonstrates how to learn rationales in one modality from another through cross-modal rationale transfer, which saves annotation effort. Finally, tweets are classified based on extracted rationales. Experiments are conducted over CrisisMMD benchmark dataset, and results show that our proposed method boosts the classification Macro-F1 by 2-35% while extracting accurate text tokens and image patches as rationales. Human evaluation also supports the claim that our proposed method is able to retrieve better image rationale patches (12%) that help to identify humanitarian classes. Our method adapts well to new, unseen datasets in zero-shot mode, achieving an accuracy of 80%.
Paper Structure (30 sections, 6 equations, 5 figures, 12 tables)

This paper contains 30 sections, 6 equations, 5 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Tweet Classification
  4. Model Interpretability
  5. Interpretable methods in crisis domain
  6. Data Collection
  7. Multimodal Dataset
  8. Rationale Collection
  9. Methodology
  10. Overview
  11. Rationale Extraction
  12. Tweet classification
  13. Experimental Setup
  14. Baselines
  15. Unimodal Text Classification Models
  16. ...and 15 more sections

Figures (5)

  • Figure 1: An example of a tweet about 'infrastructure damage'. Highlighted words and unmasked image patches are rationales.
  • Figure 2: VLTCrisis architecture. GRU and FC are Gated Recurrent Unit and Fully Connected Layer, respectively.
  • Figure 3: Example of an image masked by a rationale heatmap.
  • Figure 4: Histogram of absolute alignment score differences between Qwen-VL and LLaVA. A difference of 0 indicates exact agreement.
  • Figure 5: Examples of predicted labels and rationales. Rationale words are highlighted.