Graph Neural Prompting with Large Language Models

TL;DR

This work tackles the limitations of large language models in grounded knowledge by introducing Graph Neural Prompting (GNP), a plug-and-play framework that retrieves subgraphs from knowledge graphs and encodes them into a trainable soft prompt fed to pre-trained LLMs. GNP uses a GNN encoder to capture graph structure, a cross-modality pooling module to align graph and text, a domain projector to map embeddings into the LLM space, and a self-supervised link-prediction objective to refine relational understanding. Experiments on commonsense and biomedical reasoning with multiple LLM sizes show that GNP provides substantial gains in frozen-LM settings and competitive improvements when tuned, often matching or surpassing full fine-tuning. The results demonstrate that instance-level, graph-informed prompts can effectively inject grounded knowledge into LLMs without large-scale retraining, enabling scalable knowledge-enhanced reasoning across domains.

Abstract

Large language models (LLMs) have shown remarkable generalization capability with exceptional performance in various language modeling tasks. However, they still exhibit inherent limitations in precisely capturing and returning grounded knowledge. While existing work has explored utilizing knowledge graphs (KGs) to enhance language modeling via joint training and customized model architectures, applying this to LLMs is problematic owing to their large number of parameters and high computational cost. Therefore, how to enhance pre-trained LLMs using grounded knowledge, e.g., retrieval-augmented generation, remains an open question. In this work, we propose Graph Neural Prompting (GNP), a novel plug-and-play method to assist pre-trained LLMs in learning beneficial knowledge from KGs. GNP encompasses various designs, including a standard graph neural network encoder, a cross-modality pooling module, a domain projector, and a self-supervised link prediction objective. Extensive experiments on multiple datasets demonstrate the superiority of GNP on both commonsense and biomedical reasoning tasks across different LLM sizes and settings. Code is available at https://github.com/meettyj/GNP.

Figures (5)

• Figure 1: Result comparison across LLM Frozen (parameters unchanged) and LLM Tuned (parameters updated) settings. The proposed Graph Neural Prompting significantly improves the performance. Reported results are averaged across six datasets on two tasks for an 11B FLAN-T5 model.
• Figure 2: The overall framework. Given a multiple choice question, we first retrieve subgraphs from the knowledge graph based on the entities in the question and options. We then develop Graph Neural Prompting (GNP) to encode the pertinent factual knowledge and structural information to obtain the Graph Neural Prompt. GNP contains various designs including a GNN, a cross-modality pooling module, a domain projector, and a self-supervised link prediction objective. Later, the obtained Graph Neural Prompt is sent into LLM for inference along with the input text embedding. We utilize the standard maximum likelihood objective for downstream task adaptation, while LLM is kept frozen or tuned depending on different experimental settings.
• Figure 3: Performance w.r.t. different number of GNN layers.
• Figure 4: Performance w.r.t. different number of cross-modality pooling layers.
• Figure 5: Case study on two QA examples from OBQA dataset. Question entities are marked in green and their subsampled neighbors in the KG are marked in blue. The entities appearing in the correct answer are marked in orange.

Theorems & Definitions (1)

• Definition 1