GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs

TL;DR

MMAGs couple text and image data with graph topology, creating a challenging setting for generation. GraphGPT-o integrates a graph-aware encoding pipeline with a DreamLLM backbone, employing Personalized PageRank sampling and either linear or hierarchical graph tokenization to produce coherent text and image outputs conditioned on local subgraphs. The framework introduces node level Q-Former and graph level Q-Former modules to capture hierarchical modality dependencies, and supports sequential and parallel inference strategies to balance modality interdependence. Across ART500K, Amazon-Baby, and Amazon-Beauty, GraphGPT-o achieves superior CLIP-based alignment and sub-graph consistency, validating its effectiveness for MMAG based generation and its potential for domains like art and e-commerce.

Abstract

The rapid development of Multimodal Large Language Models (MLLMs) has enabled the integration of multiple modalities, including texts and images, within the large language model (LLM) framework. However, texts and images are usually interconnected, forming a multimodal attributed graph (MMAG). It is underexplored how MLLMs can incorporate the relational information (\textit{i.e.}, graph structure) and semantic information (\textit{i.e.,} texts and images) on such graphs for multimodal comprehension and generation. In this paper, we propose GraphGPT-o, which supports omni-multimodal understanding and creation on MMAGs. We first comprehensively study linearization variants to transform semantic and structural information as input for MLLMs. Then, we propose a hierarchical aligner that enables deep graph encoding, bridging the gap between MMAGs and MLLMs. Finally, we explore the inference choices, adapting MLLM to interleaved text and image generation in graph scenarios. Extensive experiments on three datasets from different domains demonstrate the effectiveness of our proposed method. Datasets and codes will be open-sourced upon acceptance.

Figures (6)

• Figure 1: The overall framework of the proposed GraphGPT-o is as follows. Given a target node in a multimodal attribute graph (MMAG), we begin by using personalized PageRank for neighbor sampling. These sampled neighboring nodes are then fed into a Hierarchical Multimodal Aligner, which aligns text, image, and graph structure data. Each modality of a node is initially encoded and fused through multiple self-attention and cross-attention layers to produce multimodal node tokens. Subsequently, the tokens are processed by a graph structure Q-former, ultimately serving as inputs to the Multimodal LLM.
• Figure 2: Qualitative evaluation. Our method exhibits better consistency with the ground truth by better utilizing the graph information from neighboring nodes.
• Figure 3: The impact of sampling strategies. Our proposed personalized PageRank sampling strategy leads to better image-text pair.
• Figure 4: Study of GraphGPT-o generation with auxiliary node feature guidance: either image or text.
• Figure 5: Study on the different number of neighbors on ART500K dataset.

Theorems & Definitions (2)

• Definition 1
• Definition 2