All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks

TL;DR

This paper tackles scalable integration of large language models (LLMs) with graph neural networks (GNNs) by introducing E-LLaGNN, an on-demand LLM augmentation framework that selectively enriches a small subset of nodes during training. It samples high-quality neighborhoods via LLMs, enriches node texts with a diverse catalog of prompts, and aggregates using conventional GNNs under a computational budget, enabling LLM-free inference at test time. The approach is complemented by several active-node selection heuristics (PageRank, clustering distance, text length, degree distribution) to scale to graphs with millions of nodes. Empirical results on Cora, PubMed, ogbn-arxiv, and ogbn-products show that targeted augmentation yields significant gains over strong baselines and maintains robustness as GNN depth increases, while also improving gradient flow in deep networks. The framework offers a practical pathway to leverage world knowledge in graph learning without incurring prohibitive inference costs, making it attractive for industry-scale deployment.

Abstract

Graph Neural Networks (GNNs) have attracted immense attention in the past decade due to their numerous real-world applications built around graph-structured data. On the other hand, Large Language Models (LLMs) with extensive pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. In this paper, we investigate how LLMs can be leveraged in a computationally efficient fashion to benefit rich graph-structured data, a modality relatively unexplored in LLM literature. Prior works in this area exploit LLMs to augment every node features in an ad-hoc fashion (not scalable for large graphs), use natural language to describe the complex structural information of graphs, or perform computationally expensive finetuning of LLMs in conjunction with GNNs. We propose E-LLaGNN (Efficient LLMs augmented GNNs), a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph. More specifically, E-LLaGNN relies on sampling high-quality neighborhoods using LLMs, followed by on-demand neighborhood feature enhancement using diverse prompts from our prompt catalog, and finally information aggregation using message passing from conventional GNN architectures. We explore several heuristics-based active node selection strategies to limit the computational and memory footprint of LLMs when handling millions of nodes. Through extensive experiments & ablation on popular graph benchmarks of varying scales (Cora, PubMed, ArXiv, & Products), we illustrate the effectiveness of our E-LLaGNN framework and reveal many interesting capabilities such as improved gradient flow in deep GNNs, LLM-free inference ability etc.

Figures (4)

• Figure 1: Overview of our proposed E-LLaGNN Framework. E-LLaGNN relies on sampling high-quality neighborhoods using LLMs, followed by on-demand neighborhood feature enhancement using diverse prompts from our prompt catalog. Enhanced neighborhood features can be aggregated with the central node using various existing GNN aggregators (e.g., GCN, GraphSAGE, or GAT).
• Figure 2: Overall flow of our LLM-free E-LLaGNN Inference Pipeline which completetly removes LLM dependence during inference.
• Figure 3: Mean gradient flow across the layers of an 8-layer E-LLaGNN GNN backbone, with and without LLM-based neighborhood enhancement on (a) Cora, and (b) ogbn-arxiv.
• Figure 4: Performance comparison of the 2-layer E-LLaGNN framework trained using varying percentages of node augmentation with LLaMa-7b & Vicuna-7b on Cora & PubMed.