Shapley Value-based Contrastive Alignment for Multimodal Information Extraction

TL;DR

This paper tackles semantic and modality gaps in multimodal information extraction by introducing the Image-Context-Text paradigm and a Shapley-value-based contrastive alignment (Shap-CA). Shap-CA uses large multimodal models to generate bridging textual context from images, then computes context contributions via Shapley values and optimizes mutual alignment through contrastive losses for context-text and context-image pairs, complemented by an adaptive fusion module. The approach yields state-of-the-art results across four MIE datasets (MNER, MRE, MJERE), with strong ablation and analysis confirming the importance of bridging context, Shapley-based alignment, and robust context generators. Overall, Shap-CA provides a principled, efficient solution for robust multimodal information extraction with practical impact for diverse social-media analytics and knowledge extraction tasks.

Abstract

The rise of social media and the exponential growth of multimodal communication necessitates advanced techniques for Multimodal Information Extraction (MIE). However, existing methodologies primarily rely on direct Image-Text interactions, a paradigm that often faces significant challenges due to semantic and modality gaps between images and text. In this paper, we introduce a new paradigm of Image-Context-Text interaction, where large multimodal models (LMMs) are utilized to generate descriptive textual context to bridge these gaps. In line with this paradigm, we propose a novel Shapley Value-based Contrastive Alignment (Shap-CA) method, which aligns both context-text and context-image pairs. Shap-CA initially applies the Shapley value concept from cooperative game theory to assess the individual contribution of each element in the set of contexts, texts and images towards total semantic and modality overlaps. Following this quantitative evaluation, a contrastive learning strategy is employed to enhance the interactive contribution within context-text/image pairs, while minimizing the influence across these pairs. Furthermore, we design an adaptive fusion module for selective cross-modal fusion. Extensive experiments across four MIE datasets demonstrate that our method significantly outperforms existing state-of-the-art methods.

Figures (4)

• Figure 1: The semantic and modality gaps.
• Figure 2: The overall architecture of Shap-CA.
• Figure 3: Model performance on MJERE with different $\alpha$ or $\beta$
• Figure 4: Two cases of the predictions by EEGA and Shap-CA (ours).