OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graph

TL;DR

OpenMAG formalizes Multimodal-Attributed Graphs (MAGs) and presents a comprehensive benchmark to standardize evaluation across datasets, encoders, models, and tasks. It defines MAG as $\mathcal{G}= (\mathcal{V},\mathcal{E},\mathcal{X}_{\mathcal{M}})$ with modalities $\mathcal{M}=\{T,V\}$ and a four-stage pipeline (encoding, fusion, interaction, adaptation), and evaluates 8 downstream tasks using diverse metrics. The benchmark integrates 19 datasets over 6 domains, supports 16 encoders and 24 MAG models across Graph-, Multimodal-, and MLLM-enhanced paradigms, and scrutinizes five evaluation dimensions to yield 14 actionable insights. This open-source framework enables fair, large-scale comparisons and guides future MAG research toward unified, robust, and scalable solutions with practical impact in cross-modal reasoning and generation on graphs.

Abstract

Multimodal-Attributed Graph (MAG) learning has achieved remarkable success in modeling complex real-world systems by integrating graph topology with rich attributes from multiple modalities. With the rapid proliferation of novel MAG models capable of handling intricate cross-modal semantics and structural dependencies, establishing a rigorous and unified evaluation standard has become imperative. Although existing benchmarks have facilitated initial progress, they exhibit critical limitations in domain coverage, encoder flexibility, model diversity, and task scope, presenting significant challenges to fair evaluation. To bridge this gap, we present OpenMAG, a comprehensive benchmark that integrates 19 datasets across 6 domains and incorporates 16 encoders to support both static and trainable feature encoding. OpenMAG further implements a standardized library of 24 state-of-the-art models and supports 8 downstream tasks, enabling fair comparisons within a unified framework. Through systematic assessment of necessity, data quality, effectiveness, robustness, and efficiency, we derive 14 fundamental insights into MAG learning to guide future advancements. Our code is available at https://github.com/YUKI-N810/OpenMAG.

Figures (7)

• Figure 1: The overall framework of OpenMAG.
• Figure 2: Performance comparison on unimodal and multimodal settings.
• Figure 3: Impact of fine-tuning strategies.
• Figure 4: Comparison of encoding strategies.
• Figure 5: Robustness analysis on different models.