Graph4MM: Weaving Multimodal Learning with Structural Information

TL;DR

Graph4MM tackles the challenge that real multimodal data exhibit complex, multi-hop structural relationships beyond simple image-caption pairs. It introduces Hop-Diffused Attention to infuse multi-hop graph structure into self-attention and MM-QFormer to perform principled cross-modal fusion, enabling structure-guided intra- and inter-modal interactions. The framework is evaluated on generative and discriminative tasks, showing consistent improvements over large VLMs, LLMs, and prior MMG-based methods, with meaningful gains across both generation and zero-shot classification. The results, along with theoretical analyses and ablations, support the claim that treating graph topology as a guide for fusion—not as a standalone modality—yields stronger multimodal understanding and practical benefits in foundation-model contexts.

Abstract

Real-world multimodal data usually exhibit complex structural relationships beyond traditional one-to-one mappings like image-caption pairs. Entities across modalities interact in intricate ways, with images and text forming diverse interconnections through contextual dependencies and co-references. Graphs provide powerful structural information for modeling intra-modal and inter-modal relationships. However, previous works fail to distinguish multi-hop neighbors and treat the graph as a standalone modality, which fragments the overall understanding. This limitation presents two key challenges in multimodal learning: (1) integrating structural information from multi-hop neighbors into foundational models, and (2) fusing modality-specific information in a principled manner. To address these challenges, we revisit the role of graphs in multimodal learning within the era of foundation models and propose Graph4MM, a graph-based multimodal learning framework. To be specific, we introduce Hop-Diffused Attention, which integrates multi-hop structural information into self-attention through causal masking and hop diffusion. Furthermore, we design MM-QFormer, a multi-mapping querying transformer for cross-modal fusion. Through theoretical and empirical analysis, we show that leveraging structures to integrate both intra- and inter-modal interactions improves multimodal understanding beyond treating them as a standalone modality. Experiments on both generative and discriminative tasks show that Graph4MM outperforms larger VLMs, LLMs, and multimodal graph baselines, achieving a 6.93% average improvement.

Figures (7)

• Figure 1: Multimodal relationships in a document, where sections, images, captions, and page descriptions form a structured graph. The task aims to generate Section III using the neighboring context.
• Figure 2: Multimodal Graph Modeling
• Figure 3: Overview of the Graph4MM framework. Hop-Diffused Attention applies adjacency heuristics via causal masking and hop diffusion, while MM-QFormer captures fine-grained text-image interactions before passing inputs to the LLM.
• Figure 4: Validation performance across epochs in the discriminative task for Hop-Diffused and Hop-Aware methods on OPT-125M.
• Figure 5: The performance in discriminative task with OPT-125M backbone across different number of unseen classes (1, 3, 5, 7, 9).

Theorems & Definitions (11)

• Proposition 3.1
• Proposition 4.1
• Proposition 1.1
• proof
• Proposition 2.1
• proof
• Definition 3.1: Dirichlet Energy
• Proposition 3.2
• proof
• Proposition 4.1