LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models

TL;DR

PromptGFM introduces a two-module Graph Foundation Model for Text-Attributed Graphs by (1) replicating the GNN message-passing process in the text space with a prompt-based GNN (Graph Understanding Module) and (2) learning a universal language-based graph vocabulary (Graph Inference Module) that maps nodes to token sequences for transferable, OOV-free representations. The framework enables multi-instruction fine-tuning on diverse graphs and tasks, coupled with constrained decoding via a prefix-tree to maintain fidelity. Empirical results show strong improvements over baselines on node classification and generative/discriminative link prediction, and demonstrate cross-graph and cross-task transferability, including intra- and inter-domain settings and multiple pretraining tasks (ND, LD). The findings highlight the potential of LLMs as GNNs and the viability of graph vocabulary learning to build general, scalable TAG GFMs with readable instructions for fine-tuning and inference.

Abstract

Text-Attributed Graphs (TAGs), where each node is associated with text descriptions, are ubiquitous in real-world scenarios. They typically exhibit distinctive structure and domain-specific knowledge, motivating the development of a Graph Foundation Model (GFM) that generalizes across diverse graphs and tasks. Despite large efforts to integrate Large Language Models (LLMs) and Graph Neural Networks (GNNs) for TAGs, existing approaches suffer from decoupled architectures with two-stage alignment, limiting their synergistic potential. Even worse, existing methods assign out-of-vocabulary (OOV) tokens to graph nodes, leading to graph-specific semantics, token explosion, and incompatibility with task-oriented prompt templates, which hinders cross-graph and cross-task transferability. To address these challenges, we propose PromptGFM, a versatile GFM for TAGs grounded in graph vocabulary learning. PromptGFM comprises two key components: (1) Graph Understanding Module, which explicitly prompts LLMs to replicate the finest GNN workflow within the text space, facilitating seamless GNN-LLM integration and elegant graph-text alignment; (2) Graph Inference Module, which establishes a language-based graph vocabulary ensuring expressiveness, transferability, and scalability, enabling readable instructions for LLM fine-tuning. Extensive experiments demonstrate our superiority and transferability across diverse graphs and tasks. The code is available at this: https://github.com/agiresearch/PromptGFM.

Figures (7)

• Figure 1: Overview of three GNN-LLM integration paradigms for graph-text alignment: (a) GNN for LLM and (b) LLM for GNN use decoupled architectures in the embedding space, while (c) our work functions LLM as GNN in the text space.
• Figure 2: LLM-driven replication of the GNN workflow. We achieve fine-grained alignment between traditional embedding-based GNN and our prompt-based GNN.
• Figure 3: The pipeline of PromptGFM. (a) Graph Understanding Module: For arbitrary TAGs from different domains, prompt-based GNN replicates traditional embedding-based GNN workflow in the text space, generating compact node representations. (b) Graph Inference Module: We establish a unified graph vocabulary and extract language-based IDs to generate massive pure-language prompts, enabling LLM fine-tuning across graphs and tasks.
• Figure 4: An instance of graph inference module in link prediction, where language-based IDs are indexed from the graph vocabulary to generate readable instructions with task-oriented templates. A multi-instruction fine-tuning framework is employed to unify graphs and tasks and learn transferable knowledge for GFMs.
• Figure 5: Ablation studies on node classification.