Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach

TL;DR

PLANET introduces a Divide-and-Conquer Multimodal Graph Foundation Model to address two core weaknesses in existing MGFMs: insufficient modality interaction and weak modality alignment. By decoupling interactions at the embedding level with EDG (Mixture-of-Experts and topology-aware attention) and alignment at the node level with NDR (Discretized Semantic Representation Space and Text-anchored General Knowledge Loss), PLANET achieves superior representations for graph-centric and multimodal generative tasks. Theoretical analyses show EDG preserves synergistic multimodal information and DSRS accelerates alignment convergence, while empirical results across node classification, link prediction, and G2Text/G2Image generation demonstrate clear performance gains and efficient computation. This framework advances MAG understanding and offers a scalable, generalizable foundation for multimodal graph learning with strong transferability to downstream tasks.

Abstract

Graph Foundation Models (GFMs) have achieved remarkable success in generalizing across diverse domains. However, they mainly focus on Text-Attributed Graphs (TAGs), leaving Multimodal-Attributed Graphs (MAGs) largely untapped. Developing Multimodal Graph Foundation Models (MGFMs) allows for leveraging the rich multimodal information in MAGs, and extends applicability to broader types of downstream tasks. While recent MGFMs integrate diverse modality information, our empirical investigation reveals two fundamental limitations of existing MGFMs: (1)they fail to explicitly model modality interaction, essential for capturing intricate cross-modal semantics beyond simple aggregation, and (2)they exhibit sub-optimal modality alignment, which is critical for bridging the significant semantic disparity between distinct modal spaces. To address these challenges, we propose PLANET (graPh topoLogy-aware modAlity iNteraction and alignmEnT), a novel framework employing a Divide-and-Conquer strategy to decouple modality interaction and alignment across distinct granularities. At the embedding granularity, (1)Embedding-wise Domain Gating (EDG) performs local semantic enrichment by adaptively infusing topology-aware cross-modal context, achieving modality interaction. At the node granularity, (2)Node-wise Discretization Retrieval (NDR) ensures global modality alignment by constructing a Discretized Semantic Representation Space (DSRS) to bridge modality gaps. Extensive experiments demonstrate that PLANET significantly outperforms state-of-the-art baselines across diverse graph-centric and multimodal generative tasks.

Figures (6)

• Figure 1: Empirical study results. (a)&(b) Performance comparison across different modalities and architectures. (c)&(d) Stepwise enhancement results.
• Figure 2: Overall architecture of the proposed method: PLANET.
• Figure 3: Illustration of expert-driven semantic extraction within the EDG module. Taking the text modality as the target example, we observe that the target text modality (in Node 3) correlates with neighboring images (in Node 1,2) via diverse attributes. Our MoE module is designed to capture these distinct semantic patterns through specialized experts, enabling the precise extraction of effective cross-modal mutual information.
• Figure 4: G2Text generation results. We report BLEU-3, BLEU-4, ROUGE-L, and CIDEr on the Flickr30k and Grocery datasets.
• Figure 5: G2Image generation results. We report CLIP scores in red and DINOv2 scores in blue across four categories selected from the Goodreads-NC and Ele-fashion datasets.

Theorems & Definitions (4)

• Definition 3.1
• Theorem 3.2
• Definition 3.3
• Theorem 3.4