Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design

Yuanfu Sun, Zhengnan Ma, Yi Fang, Jing Ma, Qiaoyu Tan

TL;DR

This work tackles the challenge of evaluating graph reasoning capabilities of LLMs without training by introducing GraphICL, a comprehensive prompt-benchmark for Text-Attributed Graphs (TAGs). GraphICL employs a unified template with four components ($$anchor ext{-}node ext{ text}, task ext{ description}, structure ext{-}aware ext{ information}, demonstrations$$) to produce 55 prompts that encode graph structure and labeled cues for two core tasks: node classification and link prediction. Across 9 datasets and multiple LLMs (e.g., LLaMA variants and GPT-4o), GraphICL-enabled LLMs often exceed state-of-the-art supervised GNNs and graph LLMs in both in-domain and cross-domain settings, underscoring the power of prompt design to unlock graph reasoning without training. The results establish a strong baseline for graph LLM research and highlight the potential of in-context learning to advance practical graph understanding in large language models.

Abstract

The growing importance of textual and relational systems has driven interest in enhancing large language models (LLMs) for graph-structured data, particularly Text-Attributed Graphs (TAGs), where samples are represented by textual descriptions interconnected by edges. While research has largely focused on developing specialized graph LLMs through task-specific instruction tuning, a comprehensive benchmark for evaluating LLMs solely through prompt design remains surprisingly absent. Without such a carefully crafted evaluation benchmark, most if not all, tailored graph LLMs are compared against general LLMs using simplistic queries (e.g., zero-shot reasoning with LLaMA), which can potentially camouflage many advantages as well as unexpected predicaments of them. To achieve more general evaluations and unveil the true potential of LLMs for graph tasks, we introduce Graph In-context Learning (GraphICL) Benchmark, a comprehensive benchmark comprising novel prompt templates designed to capture graph structure and handle limited label knowledge. Our systematic evaluation shows that general-purpose LLMs equipped with our GraphICL outperform state-of-the-art specialized graph LLMs and graph neural network models in resource-constrained settings and out-of-domain tasks. These findings highlight the significant potential of prompt engineering to enhance LLM performance on graph learning tasks without training and offer a strong baseline for advancing research in graph LLMs.

GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Design

TL;DR

This work tackles the challenge of evaluating graph reasoning capabilities of LLMs without training by introducing GraphICL, a comprehensive prompt-benchmark for Text-Attributed Graphs (TAGs). GraphICL employs a unified template with four components () to produce 55 prompts that encode graph structure and labeled cues for two core tasks: node classification and link prediction. Across 9 datasets and multiple LLMs (e.g., LLaMA variants and GPT-4o), GraphICL-enabled LLMs often exceed state-of-the-art supervised GNNs and graph LLMs in both in-domain and cross-domain settings, underscoring the power of prompt design to unlock graph reasoning without training. The results establish a strong baseline for graph LLM research and highlight the potential of in-context learning to advance practical graph understanding in large language models.

Abstract

The growing importance of textual and relational systems has driven interest in enhancing large language models (LLMs) for graph-structured data, particularly Text-Attributed Graphs (TAGs), where samples are represented by textual descriptions interconnected by edges. While research has largely focused on developing specialized graph LLMs through task-specific instruction tuning, a comprehensive benchmark for evaluating LLMs solely through prompt design remains surprisingly absent. Without such a carefully crafted evaluation benchmark, most if not all, tailored graph LLMs are compared against general LLMs using simplistic queries (e.g., zero-shot reasoning with LLaMA), which can potentially camouflage many advantages as well as unexpected predicaments of them. To achieve more general evaluations and unveil the true potential of LLMs for graph tasks, we introduce Graph In-context Learning (GraphICL) Benchmark, a comprehensive benchmark comprising novel prompt templates designed to capture graph structure and handle limited label knowledge. Our systematic evaluation shows that general-purpose LLMs equipped with our GraphICL outperform state-of-the-art specialized graph LLMs and graph neural network models in resource-constrained settings and out-of-domain tasks. These findings highlight the significant potential of prompt engineering to enhance LLM performance on graph learning tasks without training and offer a strong baseline for advancing research in graph LLMs.
Paper Structure (27 sections, 3 figures, 16 tables)

This paper contains 27 sections, 3 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Prompt Design Driven by Graph In-Context Learning
  5. Basic Content
  6. Optional Enhancing Content
  7. Graph In-Context Learning Prompting
  8. Experiments
  9. Experiment Configurations
  10. Multi-scenario Graph Reasoning Testing
  11. In-Domain Node Classification
  12. Cross-Domain Node Classification
  13. Link Prediction Testing
  14. Impact of GraphICL Configuration
  15. LLMs Comparison with GraphICL
  16. ...and 12 more sections

Figures (3)

  • Figure 1: The overall framework of our GraphICL. We implement various graph in-context learning templates by combining basic content with optional enhancing content. These templates are then input as prompts into large language models to obtain relevant prediction results.
  • Figure 2: Examples of graph in-context learning prompting in different graph reasoning tasks.
  • Figure 3: The comparison heat map between the GraphICL method based on LLaMA3-70B-Instruct and the best Supervised GNNs results (SAGE-90.18%) in the table \ref{['tab:extra in-domain results']} in PubMed. The results of the GNNs are used as the baseline, with higher performance relative to the baseline shown in red and lower performance shown in blue.