Stepwise Schema-Guided Prompting Framework with Parameter Efficient Instruction Tuning for Multimedia Event Extraction

TL;DR

This paper tackles the dual challenges of modeling complex multimodal event structures and lacking multimodal training data for Multimedia Event Extraction (MEE). It introduces Stepwise Schema-Guided Prompting Framework (SSGPF) that uses a Multimodal LLM as backbone, decomposing MEE into MED and MEAE through ETSGP and ARSGP with a text-bridged grounding mechanism. A weakly-aligned multimodal event-labeled dataset is constructed from unimodal corpora via crossmodal retrieval and instruction-tuning-ready formats, followed by parameter-efficient LoRA tuning. Experiments on the M2E2 benchmark show SSGPF achieving state-of-the-art results, with notable improvements in both event detection and argument extraction, validating the effectiveness of schema-guided prompting and multimodal data construction. The approach paves the way for efficient adaptation of MLLMs to structured multimodal information extraction tasks.

Abstract

Multimedia Event Extraction (MEE) has become an important task in information extraction research as news today increasingly prefers to contain multimedia content. Current MEE works mainly face two challenges: (1) Inadequate extraction framework modeling for handling complex and flexible multimedia event structure; (2) The absence of multimodal-aligned training data for effective knowledge transfer to MEE task. In this work, we propose a Stepwise Schema-Guided Prompting Framework (SSGPF) using Multimodal Large Language Model (MLLM) as backbone for adaptive structure capturing to solve MEE task. At the initial step of SSGPF, we design Event Type Schema Guided Prompting (ETSGP) for event detection, then we devise Argument Role Schema Guided Prompting (ARSGP) that contains multi-step prompts with text-bridged grounding technique for argument extraction. We construct a weakly-aligned multimodal event labeled dataset based on existing unimodal event annotations, then conduct parameter efficient instruction tuning with LoRA on LLaVA-v1.5-7B under SSGPF. Experiments on the M2E2 benchmark demonstrate that SSGPF significantly outperforms current SOTA baselines by 5.8 percent F1 on event detection and 8.4 percent F1 on argument extraction.

Figures (4)

• Figure 1: Example of Multimedia Event Extraction. Given the multimodal input, model needs to extract event type, trigger word, and spot arguments belonging to each role label predefined in corresponding event type's argument schema.
• Figure 2: Overall workflow of SSGPF. The MLLM backbone consists of a vision encoder and an LLM. At the initial step, we design ETSGP to prompt MLLM to output event type and trigger word. Then we find the argument role schema of the predicted event type in event schema and design ARSGP with multi-step prompting. Each step in ARSGP focuses on one role label, creating adaptive prompt including predicted event type, trigger word, and the targeted role label to prompt MLLM to output all arguments in text and image of this role, where image argument is located by an external visual grounding model according to the generated textual description. During tuning stage, only the injected LoRA modules in LLM of MLLM are trained while all other modules are frozen.
• Figure 3: Experimental results of training set constructing strategy variants.
• Figure 4: Visualizations of SSGPF image argument extraction on M2E2 test set. Text in green color is the predicted description for the argument in image of the given targeted role label. Green bounding box in image denotes the argument's visual region outputted by the visual grounding model based on the predicted description. Orange bounding box in image is the ground truth.