Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Parameter-Efficient Tuning Large Language Models for Graph Representation Learning

Qi Zhu, Da Zheng, Xiang Song, Shichang Zhang, Bowen Jin, Yizhou Sun, George Karypis

TL;DR

This work tackles scalable graph representation learning on text-rich graphs by introducing Graph-aware Parameter-Efficient Fine-Tuning (GPEFT). It converts graph structure into a graph prompt using a GNN, prepends this prompt to node texts, and pre-trains the GNN prompt with next-token prediction before fine-tuning the model with a contrastive objective to produce node embeddings via the LLM. The approach achieves consistent improvements over strong baselines across eight graphs and is compatible with multiple LLMs such as LLaMA, OPT, and Falcon, while maintaining a small number of trainable parameters. The results demonstrate that GPEFT provides a practical, efficient pathway to leverage the capabilities of billion-parameter LLMs for industrial-scale link prediction tasks in text-rich graphs.

Abstract

Text-rich graphs, which exhibit rich textual information on nodes and edges, are prevalent across a wide range of real-world business applications. Large Language Models (LLMs) have demonstrated remarkable abilities in understanding text, which also introduced the potential for more expressive modeling in text-rich graphs. Despite these capabilities, efficiently applying LLMs to representation learning on graphs presents significant challenges. Recently, parameter-efficient fine-tuning methods for LLMs have enabled efficient new task generalization with minimal time and memory consumption. Inspired by this, we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning with LLMs on text-rich graphs. Specifically, we utilize a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. This prompt is then inserted at the beginning of the text sequence. To improve the quality of graph prompts, we pre-trained the GNN to assist the frozen LLM in predicting the next token in the node text. Compared with existing joint GNN and LMs, our method directly generate the node embeddings from large language models with an affordable fine-tuning cost. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations. Our results demonstrate the efficacy and efficiency of our model, showing that it can be smoothly integrated with various large language models, including OPT, LLaMA and Falcon.

Parameter-Efficient Tuning Large Language Models for Graph Representation Learning

TL;DR

This work tackles scalable graph representation learning on text-rich graphs by introducing Graph-aware Parameter-Efficient Fine-Tuning (GPEFT). It converts graph structure into a graph prompt using a GNN, prepends this prompt to node texts, and pre-trains the GNN prompt with next-token prediction before fine-tuning the model with a contrastive objective to produce node embeddings via the LLM. The approach achieves consistent improvements over strong baselines across eight graphs and is compatible with multiple LLMs such as LLaMA, OPT, and Falcon, while maintaining a small number of trainable parameters. The results demonstrate that GPEFT provides a practical, efficient pathway to leverage the capabilities of billion-parameter LLMs for industrial-scale link prediction tasks in text-rich graphs.

Abstract

Text-rich graphs, which exhibit rich textual information on nodes and edges, are prevalent across a wide range of real-world business applications. Large Language Models (LLMs) have demonstrated remarkable abilities in understanding text, which also introduced the potential for more expressive modeling in text-rich graphs. Despite these capabilities, efficiently applying LLMs to representation learning on graphs presents significant challenges. Recently, parameter-efficient fine-tuning methods for LLMs have enabled efficient new task generalization with minimal time and memory consumption. Inspired by this, we introduce Graph-aware Parameter-Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning with LLMs on text-rich graphs. Specifically, we utilize a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. This prompt is then inserted at the beginning of the text sequence. To improve the quality of graph prompts, we pre-trained the GNN to assist the frozen LLM in predicting the next token in the node text. Compared with existing joint GNN and LMs, our method directly generate the node embeddings from large language models with an affordable fine-tuning cost. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations. Our results demonstrate the efficacy and efficiency of our model, showing that it can be smoothly integrated with various large language models, including OPT, LLaMA and Falcon.
Paper Structure (19 sections, 2 theorems, 5 equations, 4 figures, 6 tables, 1 algorithm)

This paper contains 19 sections, 2 theorems, 5 equations, 4 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Notations and Preliminaries
  4. Text-Rich Graph Representation Learning
  5. Casual Language Modeling
  6. Parameter-Efficient LLM Fine-tuning
  7. Method
  8. Graph prompt tuning
  9. Graph representation learning with LLMs
  10. Model Optimization.
  11. Experiments
  12. Datasets
  13. Baselines
  14. Experiment Settings
  15. Experiment Results
  16. ...and 4 more sections

Key Result

theorem 1

Consider the $l$-th layer of a transformer, where $H^l_k = [h^l_{i,0}, \ldots, h^l_{i,k}]$ represents the embedding matrix for a text sequence of length $k$. At each layer, appending a graph prompt to the sequence results in a convex combination of the embeddings $[H^l_k; z_i]$. In contrast, prepend

Figures (4)

  • Figure 1: Convenient use case of parameter-efficient fine-tuning for text-rich graphs, requiring only 2% additional LLM parameters per graph application.
  • Figure 2: Overview of the GPEFT Framework: Architecture (Left) and Pre-training and Fine-tuning Processes (Right).
  • Figure 3: GPEFT using different LLMs and PEFT algorithms on Amazon Review.
  • Figure 4: GPEFT using different LLMs and PEFT algorithms on Amazon Review.

Theorems & Definitions (2)

  • theorem 1
  • theorem 2