Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

Yijian Qin, Xin Wang, Ziwei Zhang, Wenwu Zhu

TL;DR

<p>The paper tackles prediction on text-attributed graphs (TAGs) by bridging structured graph information and large language models (LLMs). It introduces the Disentangled Graph Text Learner (DGTL), which uses multiple disentangled GNN channels to encode multi-factor neighborhood information and injects these signals into a frozen LLM, balancing performance with efficiency and stability. DGTL demonstrates competitive results against state-of-the-art baselines on TAG benchmarks and provides natural language explanations that articulate the graph-driven reasoning behind predictions. This approach offers a practical pathway to leverage LLMs for TAG tasks with interpretable, cost-efficient reasoning in domains like citation and e-commerce networks.

Abstract

Text-attributed graphs (TAGs) are prevalent on the web and research over TAGs such as citation networks, e-commerce networks and social networks has attracted considerable attention in the web community. Recently, large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks. However, the existing works focus on harnessing the potential of LLMs solely relying on prompts to convey graph structure information to LLMs, thus suffering from insufficient understanding of the complex structural relationships within TAGs. To address this problem, in this paper we present the Disentangled Graph-Text Learner (DGTL) model, which is able to enhance the reasoning and predicting capabilities of LLMs for TAGs. Our proposed DGTL model incorporates graph structure information through tailored disentangled graph neural network (GNN) layers, enabling LLMs to capture the intricate relationships hidden in text-attributed graphs from multiple structural factors. Furthermore, DGTL operates with frozen pre-trained LLMs, reducing computational costs and allowing much more flexibility in combining with different LLM models. Experimental evaluations demonstrate the effectiveness of the proposed DGTL model on achieving superior or comparable performance over state-of-the-art baselines. Additionally, we also demonstrate that our DGTL model can offer natural language explanations for predictions, thereby significantly enhancing model interpretability.

Disentangled Representation Learning with Large Language Models for Text-Attributed Graphs

TL;DR

<p>The paper tackles prediction on text-attributed graphs (TAGs) by bridging structured graph information and large language models (LLMs). It introduces the Disentangled Graph Text Learner (DGTL), which uses multiple disentangled GNN channels to encode multi-factor neighborhood information and injects these signals into a frozen LLM, balancing performance with efficiency and stability. DGTL demonstrates competitive results against state-of-the-art baselines on TAG benchmarks and provides natural language explanations that articulate the graph-driven reasoning behind predictions. This approach offers a practical pathway to leverage LLMs for TAG tasks with interpretable, cost-efficient reasoning in domains like citation and e-commerce networks.

Abstract

Text-attributed graphs (TAGs) are prevalent on the web and research over TAGs such as citation networks, e-commerce networks and social networks has attracted considerable attention in the web community. Recently, large language models (LLMs) have demonstrated exceptional capabilities across a wide range of tasks. However, the existing works focus on harnessing the potential of LLMs solely relying on prompts to convey graph structure information to LLMs, thus suffering from insufficient understanding of the complex structural relationships within TAGs. To address this problem, in this paper we present the Disentangled Graph-Text Learner (DGTL) model, which is able to enhance the reasoning and predicting capabilities of LLMs for TAGs. Our proposed DGTL model incorporates graph structure information through tailored disentangled graph neural network (GNN) layers, enabling LLMs to capture the intricate relationships hidden in text-attributed graphs from multiple structural factors. Furthermore, DGTL operates with frozen pre-trained LLMs, reducing computational costs and allowing much more flexibility in combining with different LLM models. Experimental evaluations demonstrate the effectiveness of the proposed DGTL model on achieving superior or comparable performance over state-of-the-art baselines. Additionally, we also demonstrate that our DGTL model can offer natural language explanations for predictions, thereby significantly enhancing model interpretability.
Paper Structure (23 sections, 8 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 23 sections, 8 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Text-attributed Graphs
  4. LLMs for Graph Tasks
  5. Problem Formulation and Preliminaries
  6. Text-attributed Graphs
  7. Graph Neural Network
  8. Large Language Models
  9. Method
  10. Overall Framework
  11. Text Embedding Generation
  12. Disentangled Graph Learning
  13. Neighborhood Information Injection
  14. Fine-tuning Procedure
  15. Experiments
  16. ...and 8 more sections

Figures (2)

  • Figure 1: An illustration of using LLMs to solve paper classification task on TAGs. Top: existing methods fail to give correct answer because of lacking structural information on the TAG. Bottom: our method can predict the correct label by utilizing the structural information, i.e., the reference information in the citation graph.
  • Figure 2: An overview of our proposed DGTL method. Step 1: Generating text embedding by taking average of the features at the last layer in the upstream LLM. Step 2: Using our proposed disentangled graph learning to learn embeddings with diverse structural information. Step 3: Injecting the features with neighborhood information into the downstream LLM.