Graph Machine Learning in the Era of Large Language Models (LLMs)

TL;DR

The paper surveys Graph ML in the era of Large Language Models, detailing how LLMs can enhance graph feature quality, reduce data requirements, and improve generalization, while graphs can in turn bolster LLM reasoning, explainability, and factual grounding. It provides a two-pronged taxonomy: LLM-enhanced Graph ML and Graph-enhanced LLMs, covering backbone architectures, self-supervised learning, pretext tasks, and prompt-based strategies. It highlights concrete applications across recommender systems, knowledge graphs, AI for science, and robotics, and identifies challenges in heterophily, OOD generalization, hallucinations, and efficiency. The work argues for Graph Foundation Models (GFMs) as a unifying direction and outlines future research on generalization, multi-modal graphs, trustworthiness, and scalable deployment.

Abstract

Graphs play an important role in representing complex relationships in various domains like social networks, knowledge graphs, and molecular discovery. With the advent of deep learning, Graph Neural Networks (GNNs) have emerged as a cornerstone in Graph Machine Learning (Graph ML), facilitating the representation and processing of graph structures. Recently, LLMs have demonstrated unprecedented capabilities in language tasks and are widely adopted in a variety of applications such as computer vision and recommender systems. This remarkable success has also attracted interest in applying LLMs to the graph domain. Increasing efforts have been made to explore the potential of LLMs in advancing Graph ML's generalization, transferability, and few-shot learning ability. Meanwhile, graphs, especially knowledge graphs, are rich in reliable factual knowledge, which can be utilized to enhance the reasoning capabilities of LLMs and potentially alleviate their limitations such as hallucinations and the lack of explainability. Given the rapid progress of this research direction, a systematic review summarizing the latest advancements for Graph ML in the era of LLMs is necessary to provide an in-depth understanding to researchers and practitioners. Therefore, in this survey, we first review the recent developments in Graph ML. We then explore how LLMs can be utilized to enhance the quality of graph features, alleviate the reliance on labeled data, and address challenges such as graph heterogeneity and out-of-distribution (OOD) generalization. Afterward, we delve into how graphs can enhance LLMs, highlighting their abilities to enhance LLM pre-training and inference. Furthermore, we investigate various applications and discuss the potential future directions in this promising field.

Figures (5)

• Figure 1: Illustration of the application of Large Language Models (LLMs) in graph machine learning. The integration of LLMs with Graph Neural Networks (GNNs) is utilized to model an extensive range of graph data across various downstream tasks.
• Figure 2: The outline of our survey. Section \ref{['sec:graph foundation model']} Deep Learning on Graphs explores the development of DNN-based methods, focusing on the Backbone Architecture, Graph Pretext Tasks, and Downstream Adaption three aspects. Section \ref{['sec:LLM-enhanced Graph']} LLMs for Graph Models explore how current LLMs help the current Graph ML towards GFMs from Enhancing Feature Quality, Solving Vanilla GNN Training Limitations, and Heterophily and Generalization three aspects. Section \ref{['sec:Graph-enhanced LLMs']} Graph for LLMs focuses on Knowledge Graph(KG)-enhanced LLM Pre-training and KG-enhanced LLM Inference. Section \ref{['sec:application']} Applications presents various applications, including Recommender System, Knowledge Graph, AI for Science, and Robot Task Planning. Section \ref{['sec:future_work']} Future Directions discusses potential future directions for LLMs in graph machine learning from the Generalization and Transferability, Multi-modal Graph Learning, Trustworthiness and Efficiency.
• Figure 3: A comparison of pre-training, fine-tuning, and prompt tuning. (a) Pre-training involves training the GNN model based on specific pre-training tasks. (b) Fine-tuning updates the parameters of the pre-trained GNN model according to the downstream tasks. (c) Prompt tuning generates and updates the features of the prompt according to the downstream tasks, while keeping the pre-trained GNN model fixed and without any modification.
• Figure 4: Illustration of LLMs for Graph ML. (1) Methods using LLMs for Enhancing Feature Quality by enhancing feature representation, generating augmented information, and aligning feature space. (2) Explorations for solving Vanilla GNN Training Limitations are categorized based on how structural information in the graph is processed: ignoring structural information, implicit structural information, and explicit structural information. (3) Research about employing LLMs to alleviate the limitations of Heterophily and Generalization.
• Figure 5: The illustration of employing LLMs with implicit and explicit structural information. (1) Methods leveraging implicit structural information describe nodes and graph structure information in natural language and combine task-specific instructions to form a textual prompt, which is then input into the LLM to generate prediction results. (2) Methods employing explicit structural information use GNNs and LLMs to encode graph and instruction information separately. Then, fusion layers are added to align the graph and text modalities, and the fused embedding is input into the LLM for prediction.