Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning

Yihan Zhang, Ercan E. Kuruoglu

TL;DR

This paper tackles modality confusion in multimodal graphs by introducing Node Splitting Graphs (NSG), which decompose each multimodal node into unimodal sub-nodes and rewire edges to enable modality-aware propagation. It then enhances expressiveness with GraphMixture-of-Experts (GraphMoE), routing modality-specific messages through specialized HGNN experts and employing a dual-branch gating mechanism with noise for balanced specialization. The authors provide spectral and information-theoretic analyses showing NSG yields adaptive, cross-modal low-pass filtering and tighter generalization bounds due to restricted parameterization. Empirically, NSG-MoE consistently outperforms strong baselines on node classification and link prediction across multiple multimodal benchmarks, while maintaining competitive training efficiency thanks to sparsification and modular expert design.

Abstract

Multimodal graphs are gaining increasing attention due to their rich representational power and wide applicability, yet they introduce substantial challenges arising from severe modality confusion. To address this issue, we propose NSG (Node Splitting Graph)-MoE, a multimodal graph learning framework that integrates a node-splitting and graph-rewiring mechanism with a structured Mixture-of-Experts (MoE) architecture. It explicitly decomposes each node into modality-specific components and assigns relation-aware experts to process heterogeneous message flows, thereby preserving structural information and multimodal semantics while mitigating the undesirable mixing effects commonly observed in general-purpose GNNs. Extensive experiments on three multimodal benchmarks demonstrate that NSG-MoE consistently surpasses strong baselines. Despite incorporating MoE -- which is typically computationally heavy -- our method achieves competitive training efficiency. Beyond empirical results, we provide a spectral analysis revealing that NSG performs adaptive filtering over modality-specific subspaces, thus explaining its disentangling behavior. Furthermore, an information-theoretic analysis shows that the architectural constraints imposed by NSG reduces mutual information between data and parameters and improving generalization capability.

Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning

TL;DR

This paper tackles modality confusion in multimodal graphs by introducing Node Splitting Graphs (NSG), which decompose each multimodal node into unimodal sub-nodes and rewire edges to enable modality-aware propagation. It then enhances expressiveness with GraphMixture-of-Experts (GraphMoE), routing modality-specific messages through specialized HGNN experts and employing a dual-branch gating mechanism with noise for balanced specialization. The authors provide spectral and information-theoretic analyses showing NSG yields adaptive, cross-modal low-pass filtering and tighter generalization bounds due to restricted parameterization. Empirically, NSG-MoE consistently outperforms strong baselines on node classification and link prediction across multiple multimodal benchmarks, while maintaining competitive training efficiency thanks to sparsification and modular expert design.

Abstract

Multimodal graphs are gaining increasing attention due to their rich representational power and wide applicability, yet they introduce substantial challenges arising from severe modality confusion. To address this issue, we propose NSG (Node Splitting Graph)-MoE, a multimodal graph learning framework that integrates a node-splitting and graph-rewiring mechanism with a structured Mixture-of-Experts (MoE) architecture. It explicitly decomposes each node into modality-specific components and assigns relation-aware experts to process heterogeneous message flows, thereby preserving structural information and multimodal semantics while mitigating the undesirable mixing effects commonly observed in general-purpose GNNs. Extensive experiments on three multimodal benchmarks demonstrate that NSG-MoE consistently surpasses strong baselines. Despite incorporating MoE -- which is typically computationally heavy -- our method achieves competitive training efficiency. Beyond empirical results, we provide a spectral analysis revealing that NSG performs adaptive filtering over modality-specific subspaces, thus explaining its disentangling behavior. Furthermore, an information-theoretic analysis shows that the architectural constraints imposed by NSG reduces mutual information between data and parameters and improving generalization capability.
Paper Structure (26 sections, 1 theorem, 31 equations, 7 figures, 7 tables, 1 algorithm)

This paper contains 26 sections, 1 theorem, 31 equations, 7 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Graph Learning
  4. Heterogeneous Graphs
  5. Graph Mixture of Experts
  6. Preliminaries
  7. Node Splitting and Graph Rewiring
  8. Maintaining scalability when the number of modalities is large
  9. Preliminary verification
  10. Theoretical Analyses
  11. A Coalesced Form of HGNNs
  12. Spectral Analyses of NSG
  13. Comparing With General GNN Models
  14. Cost-effective Mixture of HGNN Experts
  15. Dual-Branch Expert Design
  16. ...and 11 more sections

Key Result

Lemma 5.1

For a model with parameters $W$ trained on sample set $S$, where $\sigma^2$ is a sub-Gaussian constant of the loss function.

Figures (7)

  • Figure 1: Intuitive understanding of the proposed node splitting mechanism. Different colors represent different modalities. Data source: zhu2025mosaic.
  • Figure 2: Illustration of node splitting and edge construction
  • Figure 3: Procedure diagram of multimodal graph learning with NSG
  • Figure 4: Overall framework of the GMoE-extended model
  • Figure 5: Analysis on the gating behavior. (node classification on Ele-fashion with CLIP; The same applies unless otherwise specified.)
  • ...and 2 more figures

Theorems & Definitions (1)

  • Lemma 5.1: Information-theoretic generalization bound