Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction

Meishan Zhang, Hao Fei, Bin Wang, Shengqiong Wu, Yixin Cao, Fei Li, Min Zhang

TL;DR

Grounded Multimodal Universal Information Extraction (MUIE) addresses the fragmentation of IE tasks by proposing a unified framework that outputs textual IE labels and fine-grained grounding across modalities. The authors introduce Reamo, a Vicuna-based multimodal LLM that leverages ImageBind for encoding and SEEM/SHAS-based grounding, and train it via UIE instruction tuning, multimodal alignment learning, grounding-aware tuning, and invocation-based meta-response tuning. They also curate a high-quality benchmark with 3,000 test instances across nine modality combinations for NER, RE, and EE, enabling comprehensive zero-shot evaluation. Experiments show Reamo outperforms existing MLLMs in end-to-end UIE and grounding across image, audio, video, and complex modality mixtures, establishing a strong early benchmark for grounded MUIE. Overall, the work advances practical, unified multimodal IE and provides publicly released resources to spur future research.

Abstract

In the field of information extraction (IE), tasks across a wide range of modalities and their combinations have been traditionally studied in isolation, leaving a gap in deeply recognizing and analyzing cross-modal information. To address this, this work for the first time introduces the concept of grounded Multimodal Universal Information Extraction (MUIE), providing a unified task framework to analyze any IE tasks over various modalities, along with their fine-grained groundings. To tackle MUIE, we tailor a multimodal large language model (MLLM), Reamo, capable of extracting and grounding information from all modalities, i.e., recognizing everything from all modalities at once. Reamo is updated via varied tuning strategies, equipping it with powerful capabilities for information recognition and fine-grained multimodal grounding. To address the absence of a suitable benchmark for grounded MUIE, we curate a high-quality, diverse, and challenging test set, which encompasses IE tasks across 9 common modality combinations with the corresponding multimodal groundings. The extensive comparison of Reamo with existing MLLMs integrated into pipeline approaches demonstrates its advantages across all evaluation dimensions, establishing a strong benchmark for the follow-up research. Our resources are publicly released at https://haofei.vip/MUIE.

Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction

TL;DR

Grounded Multimodal Universal Information Extraction (MUIE) addresses the fragmentation of IE tasks by proposing a unified framework that outputs textual IE labels and fine-grained grounding across modalities. The authors introduce Reamo, a Vicuna-based multimodal LLM that leverages ImageBind for encoding and SEEM/SHAS-based grounding, and train it via UIE instruction tuning, multimodal alignment learning, grounding-aware tuning, and invocation-based meta-response tuning. They also curate a high-quality benchmark with 3,000 test instances across nine modality combinations for NER, RE, and EE, enabling comprehensive zero-shot evaluation. Experiments show Reamo outperforms existing MLLMs in end-to-end UIE and grounding across image, audio, video, and complex modality mixtures, establishing a strong early benchmark for grounded MUIE. Overall, the work advances practical, unified multimodal IE and provides publicly released resources to spur future research.

Abstract

In the field of information extraction (IE), tasks across a wide range of modalities and their combinations have been traditionally studied in isolation, leaving a gap in deeply recognizing and analyzing cross-modal information. To address this, this work for the first time introduces the concept of grounded Multimodal Universal Information Extraction (MUIE), providing a unified task framework to analyze any IE tasks over various modalities, along with their fine-grained groundings. To tackle MUIE, we tailor a multimodal large language model (MLLM), Reamo, capable of extracting and grounding information from all modalities, i.e., recognizing everything from all modalities at once. Reamo is updated via varied tuning strategies, equipping it with powerful capabilities for information recognition and fine-grained multimodal grounding. To address the absence of a suitable benchmark for grounded MUIE, we curate a high-quality, diverse, and challenging test set, which encompasses IE tasks across 9 common modality combinations with the corresponding multimodal groundings. The extensive comparison of Reamo with existing MLLMs integrated into pipeline approaches demonstrates its advantages across all evaluation dimensions, establishing a strong benchmark for the follow-up research. Our resources are publicly released at https://haofei.vip/MUIE.
Paper Structure (43 sections, 2 equations, 7 figures, 6 tables)

This paper contains 43 sections, 2 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Task Definition: Grounded Multimodal Universal Information Extraction
  4. Our Proposed Model
  5. MLLM Framework of Reamo
  6. Multimodal Encoding.
  7. LLM Reasoner.
  8. MUIE Decoding with Grounding.
  9. MUIE Fine-tuning for Reamo
  10. UIE Instruction Tuning.
  11. Multimodal Alignment Learning.
  12. Fine-grained Cross-modal Grounding-aware Tuning.
  13. Invocation-based Meta-response Tuning.
  14. A Benchmark for Grounded MUIE
  15. Data Source
  16. ...and 28 more sections

Figures (7)

  • Figure 1: Examples of grounded multimodal universal information extraction (MUIE).
  • Figure 2: An overview of the proposed Reamo MLLM architecture for grounded MUIE.
  • Figure 3: Performance gap between modality-shared (aligned) and modality-specific (unaligned) MUIE.
  • Figure 4: Impact of different object/entity numbers.
  • Figure 5: Qualitative result A on MUIE (NER) with modality-specific case via reasoning.
  • ...and 2 more figures