Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Can we Soft Prompt LLMs for Graph Learning Tasks?

Zheyuan Liu, Xiaoxin He, Yijun Tian, Nitesh V. Chawla

TL;DR

The paper tackles whether large language models can handle graph learning tasks when guided by soft prompts. It introduces GraphPrompter, which fuses a GNN-based graph encoder with a frozen LLM by aligning per-node graph embeddings with textual attributes to form a soft prompt for node classification and link prediction. Empirical results across five graph benchmarks show strong performance, especially when using LoRA to tune the LLM, while zero-shot configurations underperform, underscoring the importance of explicit graph-aware prompting. Overall, the work demonstrates a viable pathway for deploying LLMs as predictors in graph contexts, offering a plug-and-play framework that blends structural and textual information for advanced graph understanding.

Abstract

Graph plays an important role in representing complex relationships in real-world applications such as social networks, biological data and citation networks. In recent years, Large Language Models (LLMs) have achieved tremendous success in various domains, which makes applying LLMs to graphs particularly appealing. However, directly applying LLMs to graph modalities presents unique challenges due to the discrepancy and mismatch between the graph and text modalities. Hence, to further investigate LLMs' potential for comprehending graph information, we introduce GraphPrompter, a novel framework designed to align graph information with LLMs via soft prompts. Specifically, GraphPrompter consists of two main components: a graph neural network to encode complex graph information and an LLM that effectively processes textual information. Comprehensive experiments on various benchmark datasets under node classification and link prediction tasks demonstrate the effectiveness of our proposed method. The GraphPrompter framework unveils the substantial capabilities of LLMs as predictors in graph-related tasks, enabling researchers to utilize LLMs across a spectrum of real-world graph scenarios more effectively.

Can we Soft Prompt LLMs for Graph Learning Tasks?

TL;DR

The paper tackles whether large language models can handle graph learning tasks when guided by soft prompts. It introduces GraphPrompter, which fuses a GNN-based graph encoder with a frozen LLM by aligning per-node graph embeddings with textual attributes to form a soft prompt for node classification and link prediction. Empirical results across five graph benchmarks show strong performance, especially when using LoRA to tune the LLM, while zero-shot configurations underperform, underscoring the importance of explicit graph-aware prompting. Overall, the work demonstrates a viable pathway for deploying LLMs as predictors in graph contexts, offering a plug-and-play framework that blends structural and textual information for advanced graph understanding.

Abstract

Graph plays an important role in representing complex relationships in real-world applications such as social networks, biological data and citation networks. In recent years, Large Language Models (LLMs) have achieved tremendous success in various domains, which makes applying LLMs to graphs particularly appealing. However, directly applying LLMs to graph modalities presents unique challenges due to the discrepancy and mismatch between the graph and text modalities. Hence, to further investigate LLMs' potential for comprehending graph information, we introduce GraphPrompter, a novel framework designed to align graph information with LLMs via soft prompts. Specifically, GraphPrompter consists of two main components: a graph neural network to encode complex graph information and an LLM that effectively processes textual information. Comprehensive experiments on various benchmark datasets under node classification and link prediction tasks demonstrate the effectiveness of our proposed method. The GraphPrompter framework unveils the substantial capabilities of LLMs as predictors in graph-related tasks, enabling researchers to utilize LLMs across a spectrum of real-world graph scenarios more effectively.
Paper Structure (13 sections, 4 equations, 2 figures, 2 tables)

This paper contains 13 sections, 4 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Method
  3. Graph Section
  4. LLM Section
  5. Experiments
  6. Experiment setups
  7. Datasets and baseline models.
  8. Implementation Details.
  9. Experiments Result
  10. Node Classification Task
  11. Link Prediction Task
  12. Analysis
  13. Conclusion

Figures (2)

  • Figure 1: Illustration of the proposed GraphPrompter for node classification task. The process involves extracting a k-hop subgraph for each node, feeding it into a GNN followed by a projection layer. Simultaneously, the textual attributes associated with each node are processed by the text embedder. The resulting node embedding is then concatenated with text embeddings, serving as a soft prompt to guide the LLM for graph learning tasks.
  • Figure 2: Node classification experiments comparing the proposed GraphPrompter with baseline methods (i.e., soft prompt tuning and fine-tuning). The $x$ axis shows dataset types, and the $y$ axis displays accuracy scores. Figure \ref{['fig:dense']} illustrates a dense semantic setting including both the title and abstract of a paper within the node embeddings, while Figure \ref{['fig:sparse']} illustrates a sparse semantic setting with title only.