Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MMUTF: Multimodal Multimedia Event Argument Extraction with Unified Template Filling

Philipp Seeberger, Dominik Wagner, Korbinian Riedhammer

TL;DR

This work introduces a unified template filling model that connects the textual and visual modalities via textual prompts and enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics in EAE.

Abstract

With the advancement of multimedia technologies, news documents and user-generated content are often represented as multiple modalities, making Multimedia Event Extraction (MEE) an increasingly important challenge. However, recent MEE methods employ weak alignment strategies and data augmentation with simple classification models, which ignore the capabilities of natural language-formulated event templates for the challenging Event Argument Extraction (EAE) task. In this work, we focus on EAE and address this issue by introducing a unified template filling model that connects the textual and visual modalities via textual prompts. This approach enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics. Experiments on the M2E2 benchmark demonstrate the effectiveness of our approach. Our system surpasses the current SOTA on textual EAE by +7% F1, and performs generally better than the second-best systems for multimedia EAE.

MMUTF: Multimodal Multimedia Event Argument Extraction with Unified Template Filling

TL;DR

This work introduces a unified template filling model that connects the textual and visual modalities via textual prompts and enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics in EAE.

Abstract

With the advancement of multimedia technologies, news documents and user-generated content are often represented as multiple modalities, making Multimedia Event Extraction (MEE) an increasingly important challenge. However, recent MEE methods employ weak alignment strategies and data augmentation with simple classification models, which ignore the capabilities of natural language-formulated event templates for the challenging Event Argument Extraction (EAE) task. In this work, we focus on EAE and address this issue by introducing a unified template filling model that connects the textual and visual modalities via textual prompts. This approach enables the exploitation of cross-ontology transfer and the incorporation of event-specific semantics. Experiments on the M2E2 benchmark demonstrate the effectiveness of our approach. Our system surpasses the current SOTA on textual EAE by +7% F1, and performs generally better than the second-best systems for multimedia EAE.
Paper Structure (27 sections, 1 equation, 3 figures, 7 tables)

This paper contains 27 sections, 1 equation, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Method
  4. Multimodal Unified Template Filling
  5. Candidate Representations
  6. Query Representations
  7. Mapping Network
  8. Training and Inference
  9. Experiments
  10. Datasets
  11. Metrics
  12. Baselines
  13. Experimental Setup
  14. Results
  15. Analysis
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Example of Multimedia Event Extraction from the M2E2 benchmark, where event argument roles are extracted from both the textual and visual modality.
  • Figure 2: The overall architecture of MMUTF. Given a text or image event, our model encodes the corresponding event template and argument roles as a textual prompt. The textual or image context interacts with the prompt via the cross-attention mechanism, resulting into candidate and query representation vectors. Here, the candidates correspond to entities or objects, depending on the modality. A matching score then assigns the candidates to the argument roles based on a predefined threshold.
  • Figure 3: EAE F1 results for our MMUTF model with varying thresholds for each modality.