Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Each Graph is a New Language: Graph Learning with LLMs

Huachi Zhou, Jiahe Du, Chuang Zhou, Chang Yang, Yilin Xiao, Yuxuan Xie, Xiao Huang

TL;DR

GDL4LLM reframes graph learning as language learning by constructing a graph-language corpus and pre-training LLMs to understand graph structure. It then fine-tunes on sampled subgraphs encoded as graph sentences to efficiently model multi-order neighborhood information, complemented by optional textual node attributes. The approach outperforms description-based and textual-attribute baselines across three real-world datasets, with notable gains from pre-training and attribute integration, and benefits from using more capable backbones (e.g., Llama-3). The method achieves strong performance with reduced token usage and faster inference, highlighting a practical path for scalable, high-order graph learning with LLMs.

Abstract

Recent efforts leverage Large Language Models (LLMs) for modeling text-attributed graph structures in node classification tasks. These approaches describe graph structures for LLMs to understand or aggregate LLM-generated textual attribute embeddings through graph structure. However, these approaches face two main limitations in modeling graph structures with LLMs. (i) Graph descriptions become verbose in describing high-order graph structure. (ii) Textual attributes alone do not contain adequate graph structure information. It is challenging to model graph structure concisely and adequately with LLMs. LLMs lack built-in mechanisms to model graph structures directly. They also struggle with complex long-range dependencies between high-order nodes and target nodes. Inspired by the observation that LLMs pre-trained on one language can achieve exceptional performance on another with minimal additional training, we propose \textbf{G}raph-\textbf{D}efined \textbf{L}anguage for \textbf{L}arge \textbf{L}anguage \textbf{M}odel (GDL4LLM). This novel framework enables LLMs to transfer their powerful language understanding capabilities to graph-structured data. GDL4LLM translates graphs into a graph language corpus instead of graph descriptions and pre-trains LLMs on this corpus to adequately understand graph structures. During fine-tuning, this corpus describes the structural information of target nodes concisely with only a few tokens. By treating graphs as a new language, GDL4LLM enables LLMs to model graph structures adequately and concisely for node classification tasks. Extensive experiments on three real-world datasets demonstrate that GDL4LLM outperforms description-based and textual attribute embeddings-based baselines by efficiently modeling different orders of graph structure with LLMs.

Each Graph is a New Language: Graph Learning with LLMs

TL;DR

GDL4LLM reframes graph learning as language learning by constructing a graph-language corpus and pre-training LLMs to understand graph structure. It then fine-tunes on sampled subgraphs encoded as graph sentences to efficiently model multi-order neighborhood information, complemented by optional textual node attributes. The approach outperforms description-based and textual-attribute baselines across three real-world datasets, with notable gains from pre-training and attribute integration, and benefits from using more capable backbones (e.g., Llama-3). The method achieves strong performance with reduced token usage and faster inference, highlighting a practical path for scalable, high-order graph learning with LLMs.

Abstract

Recent efforts leverage Large Language Models (LLMs) for modeling text-attributed graph structures in node classification tasks. These approaches describe graph structures for LLMs to understand or aggregate LLM-generated textual attribute embeddings through graph structure. However, these approaches face two main limitations in modeling graph structures with LLMs. (i) Graph descriptions become verbose in describing high-order graph structure. (ii) Textual attributes alone do not contain adequate graph structure information. It is challenging to model graph structure concisely and adequately with LLMs. LLMs lack built-in mechanisms to model graph structures directly. They also struggle with complex long-range dependencies between high-order nodes and target nodes. Inspired by the observation that LLMs pre-trained on one language can achieve exceptional performance on another with minimal additional training, we propose \textbf{G}raph-\textbf{D}efined \textbf{L}anguage for \textbf{L}arge \textbf{L}anguage \textbf{M}odel (GDL4LLM). This novel framework enables LLMs to transfer their powerful language understanding capabilities to graph-structured data. GDL4LLM translates graphs into a graph language corpus instead of graph descriptions and pre-trains LLMs on this corpus to adequately understand graph structures. During fine-tuning, this corpus describes the structural information of target nodes concisely with only a few tokens. By treating graphs as a new language, GDL4LLM enables LLMs to model graph structures adequately and concisely for node classification tasks. Extensive experiments on three real-world datasets demonstrate that GDL4LLM outperforms description-based and textual attribute embeddings-based baselines by efficiently modeling different orders of graph structure with LLMs.
Paper Structure (25 sections, 1 theorem, 10 equations, 4 figures, 4 tables)

This paper contains 25 sections, 1 theorem, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Methodology
  4. Graph Language Pre-training
  5. Graph Token and Graph Sentence
  6. Graph Corpus Sampling
  7. Pre-training LLMs on the Corpus
  8. Connection between Pre-training and Graph Learning
  9. Graph Structure-aware Fine-tuning
  10. Absorbing Graph Structure Knowledge
  11. Fine-tuning with Graph Language
  12. Fine-tuning with Textual Node Attributes
  13. Experiments
  14. Dataset
  15. Baselines
  16. ...and 10 more sections

Key Result

Theorem 1

For a language model with sufficient capacity to construct the inner product between all $\mathbf{W}_{h,q}$ and $\mathbf{t}_q$ pairs, given node $s_{i,q}$ with degree $d_{q}$, then $\mathbf{W}_{h,q} \cdot \mathbf{t}_q \propto \log \left(\frac{\mathbb{I}_{(s_{i,q-1}, s_{i,q}) \in \mathcal{E}} \cdot \

Figures (4)

  • Figure 1: The figure demonstrates a comparison between mainstream methods and GDL4LLM for node-classification task. Figure (a) utilizes LLMs to embed node attributes and leverages GNN to aggregate the embeddings. Figure (b) presents the descriptions of graph structure centered around target nodes. Figure (c) illustrates how LLMs are pre-trained to capture graph structures through graph language learning, and how textual attributes are further integrated to enhance LLMs fine-tuning.
  • Figure 2: Accuracy comparison of different GDL4LLM variants on the test set across three datasets.
  • Figure 3: Performance comparison between Llama-2 and Llama-3 backbones on validation and test sets across three datasets.
  • Figure 4: Visualizations of the impact of graph sentence length $l$ and graph sentence length $k$ on performance.

Theorems & Definitions (1)

  • Theorem 1