Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs

Abhishek Dey, Aabha Bothera, Samhita Sarikonda, Rishav Aryan, Sanjay Kumar Podishetty, Akshay Havalgi, Gaurav Singh, Saurabh Srivastava

TL;DR

The paper tackles real-time disaster event detection on social media, where multimodal data and noisy language challenge traditional unimodal systems. It benchmarks supervised text-only, vision-only, and multimodal fusion models against prompting-based and generative VLM/LLM baselines, and reframes ED as a generative, multimodal task capable of handling missing modalities. Key contributions include a novel multi-modal ED formulation, a perturbation analysis of social-media language, and a benchmark for robust evaluation across modalities. The findings show multimodal supervised approaches achieve state-of-the-art accuracy, while large generative models lag in precision but exhibit resilience to noisy text, informing practical disaster-monitoring system design and future work in prompt design and real-time deployment.

Abstract

In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches.

Multimodal Event Detection: Current Approaches and Defining the New Playground through LLMs and VLMs

TL;DR

The paper tackles real-time disaster event detection on social media, where multimodal data and noisy language challenge traditional unimodal systems. It benchmarks supervised text-only, vision-only, and multimodal fusion models against prompting-based and generative VLM/LLM baselines, and reframes ED as a generative, multimodal task capable of handling missing modalities. Key contributions include a novel multi-modal ED formulation, a perturbation analysis of social-media language, and a benchmark for robust evaluation across modalities. The findings show multimodal supervised approaches achieve state-of-the-art accuracy, while large generative models lag in precision but exhibit resilience to noisy text, informing practical disaster-monitoring system design and future work in prompt design and real-time deployment.

Abstract

In this paper, we study the challenges of detecting events on social media, where traditional unimodal systems struggle due to the rapid and multimodal nature of data dissemination. We employ a range of models, including unimodal ModernBERT and ConvNeXt-V2, multimodal fusion techniques, and advanced generative models like GPT-4o, and LLaVA. Additionally, we also study the effect of providing multimodal generative models (such as GPT-4o) with a single modality to assess their efficacy. Our results indicate that while multimodal approaches notably outperform unimodal counterparts, generative approaches despite having a large number of parameters, lag behind supervised methods in precision. Furthermore, we also found that they lag behind instruction-tuned models because of their inability to generate event classes correctly. During our error analysis, we discovered that common social media issues such as leet speak, text elongation, etc. are effectively handled by generative approaches but are hard to tackle using supervised approaches.
Paper Structure (25 sections, 9 equations, 4 figures, 2 tables)

This paper contains 25 sections, 9 equations, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. Supervised Approaches.
  4. Prompting Approaches.
  5. Related Works
  6. Event Detection (ED).
  7. Multi-Modal Event Detection (ED).
  8. Vision-Language and Large Language Models (VLMs/LLMs).
  9. Our Approach.
  10. Experimental Setup
  11. Dataset.
  12. Evaluation Metrics.
  13. Approaches
  14. Text-Only Model
  15. Vision-Only Model
  16. ...and 10 more sections

Figures (4)

  • Figure 1: Overview of Multi-modal Event Detection. In addition to text, the input is accompanied by images and are fed to model to identify events.
  • Figure 2: Overview of few-shot prompting strategy to perform ED. In addition to providing a few-shot in-context demonstration (ICL), we also provide the model with task instruction and output format to extract the correct event class for the test instance.
  • Figure 3: The error prediction by GPT-4o. In the first image, GPT-4o misclassified the event as a "flood" category while the input instance was tagged as an "earthquake" event. Similarly, for images 2 and 3, enough contextual signal is unavailable to deduce the correct event type.
  • Figure 4: Error Categorization of Generative Approaches.