OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transport

TL;DR

This paper addresses cross-modal misalignment in Multimodal Attributed Graphs, where edges may connect semantically similar nodes in one modality but not another, causing noisy GNN aggregations. It introduces OptiMAG, a plug-in regularizer based on Unbalanced Fused Gromov-Wasserstein (UFGW), which jointly aligns modality-specific semantic topologies with the explicit graph topology and relaxes margins via KL penalties to suppress conflicting edges. The method leverages geometric space construction from modality embeddings and PPR-smoothed graph distances, a fused transport objective with an anchor term, and an efficient subgraph Sinkhorn optimization to scale to large MAGs. Empirical results across six MAG benchmarks show consistent gains in graph-centric tasks and multimodal generation, with the largest improvements for pretrained encoders like UniGraph2, highlighting OptiMAG’s ability to steer large models toward better graph adaptation. The work provides a principled, scalable framework for mitigating structural-semantic conflicts and offers practical impact for multimodal graph learning in real-world applications.

Abstract

Multimodal Attributed Graphs (MAGs) have been widely adopted for modeling complex systems by integrating multi-modal information, such as text and images, on nodes. However, we identify a discrepancy between the implicit semantic structure induced by different modality embeddings and the explicit graph structure. For instance, neighbors in the explicit graph structure may be close in one modality but distant in another. Since existing methods typically perform message passing over the fixed explicit graph structure, they inadvertently aggregate dissimilar features, introducing modality-specific noise and impeding effective node representation learning. To address this, we propose OptiMAG, an Unbalanced Optimal Transport-based regularization framework. OptiMAG employs the Fused Gromov-Wasserstein distance to explicitly guide cross-modal structural consistency within local neighborhoods, effectively mitigating structural-semantic conflicts. Moreover, a KL divergence penalty enables adaptive handling of cross-modal inconsistencies. This framework can be seamlessly integrated into existing multimodal graph models, acting as an effective drop-in regularizer. Experiments demonstrate that OptiMAG consistently outperforms baselines across multiple tasks, ranging from graph-centric tasks (e.g., node classification, link prediction) to multimodal-centric generation tasks (e.g., graph2text, graph2image). The source code will be available upon acceptance.

Figures (5)

• Figure 1: Cross-modal inconsistency in real MAG edges. Top (ele-fashion): two connected product nodes share the "Dickies Men's" brand (textually similar) yet differ visually (T-shirt vs. cap). Bottom (Reddit-S): two connected posts show similar squirrel-with-nut imagery but diverge in text content.
• Figure 2: Overview of OptiMAG's Unbalanced Fused Gromov-Wasserstein (UFGW) alignment. Left: For each batch, we build a modality cost matrix $C_M$ (cosine distances) and a graph cost matrix $C_G$ (PPR-based diffusion distances), along with a prior alignment matrix $M$. Center: Sinkhorn iterations yield a transport plan $\pi$ that exhibits three behaviors—(i) Anchor: high-consistency nodes retain large mass; (ii) Local Smoothing: mildly misaligned nodes spread mass to neighbors; (iii) Noise Rejection: severely conflicting nodes receive negligible mass via KL penalty. Right: The regularization loss $\mathcal{L}_{\mathrm{Reg}}$ is combined with the task loss $\mathcal{L}_{\mathrm{Task}}$ to update the encoder.
• Figure 3: Hyperparameter sensitivity analysis on Reddit-S (Node Classification). Left: Impact of structure weight $\alpha$, which balances the prior anchor (Wasserstein term) and structural alignment (GW term). Performance peaks at $\alpha \approx 0.6$. Right: Impact of KL penalty $\rho$, which controls tolerance to marginal deviation. An inverted U-shape is observed, with the optimum at $\rho \approx 0.1$.
• Figure 4: Scalability analysis on Reddit-S. We report the training time per epoch w.r.t. the number of graph nodes. OptiMAG maintains linear scalability, whereas Full-Graph OT exhibits cubic complexity and triggers OOM (Out Of Memory) errors when nodes $>10k$.
• Figure 5: Empirical analysis across multiple MAG datasets. Each point represents an edge in the corresponding dataset, with 1000 edges sampled per dataset. The x-axis indicates the cosine similarity of text modality embeddings between two connected nodes, while the y-axis indicates the cosine similarity of image modality embeddings.