Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Refining Interactions: Enhancing Anisotropy in Graph Neural Networks with Language Semantics

Zhaoxing Li, Xiaoming Zhang, Haifeng Zhang, Chengxiang Liu

TL;DR

LanSAGNN addresses the challenge of leveraging LLMs for TAGs by introducing language-semantics anisotropy, where node-pair specific semantic messages are produced via a dual-layer LLM finetuning framework. The Knowledge Base LLM generates task-aligned texts, while the Extraction LLM, fine-tuned with LoRA on a graph-aware corpus, converts these texts into embeddings for GNN aggregation, optionally filtered by the Optional Edge Filter to reduce cost. The method achieves SOTA results on four public TAG datasets, notably 91.24% on Cora, 82.34% on Citeseer, 94.69% on Pubmed, and 87.56% on Wiki-CS, without increasing model complexity. This work demonstrates that aligning LLM outputs with graph structure at the language level can outperform isotropic methods and offers a practical, data-efficient path toward robust LLM–GNN pipelines, while highlighting interpretability and deployment considerations for future research.

Abstract

The integration of Large Language Models (LLMs) with Graph Neural Networks (GNNs) has recently been explored to enhance the capabilities of Text Attribute Graphs (TAGs). Most existing methods feed textual descriptions of the graph structure or neighbouring nodes' text directly into LLMs. However, these approaches often cause LLMs to treat structural information simply as general contextual text, thus limiting their effectiveness in graph-related tasks. In this paper, we introduce LanSAGNN (Language Semantic Anisotropic Graph Neural Network), a framework that extends the concept of anisotropic GNNs to the natural language level. This model leverages LLMs to extract tailor-made semantic information for node pairs, effectively capturing the unique interactions within node relationships. In addition, we propose an efficient dual-layer LLMs finetuning architecture to better align LLMs' outputs with graph tasks. Experimental results demonstrate that LanSAGNN significantly enhances existing LLM-based methods without increasing complexity while also exhibiting strong robustness against interference.

Refining Interactions: Enhancing Anisotropy in Graph Neural Networks with Language Semantics

TL;DR

LanSAGNN addresses the challenge of leveraging LLMs for TAGs by introducing language-semantics anisotropy, where node-pair specific semantic messages are produced via a dual-layer LLM finetuning framework. The Knowledge Base LLM generates task-aligned texts, while the Extraction LLM, fine-tuned with LoRA on a graph-aware corpus, converts these texts into embeddings for GNN aggregation, optionally filtered by the Optional Edge Filter to reduce cost. The method achieves SOTA results on four public TAG datasets, notably 91.24% on Cora, 82.34% on Citeseer, 94.69% on Pubmed, and 87.56% on Wiki-CS, without increasing model complexity. This work demonstrates that aligning LLM outputs with graph structure at the language level can outperform isotropic methods and offers a practical, data-efficient path toward robust LLM–GNN pipelines, while highlighting interpretability and deployment considerations for future research.

Abstract

The integration of Large Language Models (LLMs) with Graph Neural Networks (GNNs) has recently been explored to enhance the capabilities of Text Attribute Graphs (TAGs). Most existing methods feed textual descriptions of the graph structure or neighbouring nodes' text directly into LLMs. However, these approaches often cause LLMs to treat structural information simply as general contextual text, thus limiting their effectiveness in graph-related tasks. In this paper, we introduce LanSAGNN (Language Semantic Anisotropic Graph Neural Network), a framework that extends the concept of anisotropic GNNs to the natural language level. This model leverages LLMs to extract tailor-made semantic information for node pairs, effectively capturing the unique interactions within node relationships. In addition, we propose an efficient dual-layer LLMs finetuning architecture to better align LLMs' outputs with graph tasks. Experimental results demonstrate that LanSAGNN significantly enhances existing LLM-based methods without increasing complexity while also exhibiting strong robustness against interference.

Paper Structure

This paper contains 17 sections, 5 equations, 2 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Integration of LLMs and GNNs
  4. Isotropic and Anisotropic GNNs
  5. Method
  6. Preliminaries
  7. Overview of LanSAGNN
  8. Optional Edge Filter
  9. Dual-layer LLMs Finetuning Architecture
  10. Knowledge Base LLM ($LLM_{\text{KB}}$)
  11. Extraction LLM ($LLM_{\text{E}}$)
  12. Downstream Pipeline
  13. Experiment
  14. Main Experiment (RQ1)
  15. Ablation and Comparative Experiment (RQ2)
  16. ...and 2 more sections

Figures (2)

  • Figure 1: Traditional GNNs (left) versus Natural Language Level (right) isotropy and anisotropy.
  • Figure 2: The overall framework of LanSAGNN.