Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning

TL;DR

This paper tackles the challenge of applying large language models to text-attributed graphs (TAGs) by bridging graph structure with in-context learning. It introduces AskGNN, a structure-enhanced retriever powered by a graph neural network to select informative node-label exemplars, coupled with a learning-to-retrieve loop that uses LLM feedback to optimize exemplar selection. The approach demonstrates consistent improvements across node classification and extends to link prediction and conditional text generation, highlighting the method's versatility and data efficiency. By enabling LLMs to exploit graph structure without extensive fine-tuning, AskGNN offers a practical pathway for scalable graph-aware reasoning with large language models.

Abstract

Textual Attributed Graphs (TAGs) are crucial for modeling complex real-world systems, yet leveraging large language models (LLMs) for TAGs presents unique challenges due to the gap between sequential text processing and graph-structured data. We introduce AskGNN, a novel approach that bridges this gap by leveraging In-Context Learning (ICL) to integrate graph data and task-specific information into LLMs. AskGNN employs a Graph Neural Network (GNN)-powered structure-enhanced retriever to select labeled nodes across graphs, incorporating complex graph structures and their supervision signals. Our learning-to-retrieve algorithm optimizes the retriever to select example nodes that maximize LLM performance on graph. Experiments across three tasks and seven LLMs demonstrate AskGNN's superior effectiveness in graph task performance, opening new avenues for applying LLMs to graph-structured data without extensive fine-tuning.

Figures (8)

• Figure 1: Illustration of methods utilizing LLMs for graph tasks, with a focus on node classification. Our proposed method enhances structure and task understanding by retrieving insightful examples that improve the LLMs' comprehension of graph complexities.
• Figure 2: Example of node classification task, illustrating how retrieved ICL examples are integrated with the query node with the prompt.
• Figure 3: The overall framework of AskGNN, illustrating the structure-enhanced retriever based on GNNs for selecting ICL examples. The framework integrates LLM feedback to optimize the retriever, improving its ability to select relevant examples for graph-based tasks.
• Figure 4: Performance of AskGNN on different tasks, including link prediction and conditional text generation.
• Figure 5: Performance of the Qwen1.5 family across parameter sizes (7B to 72B) on ogbn-product and ogbn-arxiv datasets.