Deep Semantic Graph Learning via LLM based Node Enhancement

TL;DR

This work tackles the limitations of shallow text embeddings in text-attributed graphs by integrating Large Language Models (LLMs) to enrich node text representations. It introduces a framework where LLM-generated semantic embeddings are fed into a Graph Transformer, leveraging multi-head self-attention to aggregate neighborhood information while preserving semantic richness. Through experiments on Cora and PubMed with varying label splits, the study demonstrates that LLM-enhanced features substantially improve node classification accuracy, with the Google LLM achieving up to 81.38% on PubMed and SBERT yielding strong results on Cora. The findings indicate that the effectiveness of language-informed graph learning depends on data availability, guiding practitioners to choose architectures (e.g., Graph Transformer vs. GCN) according to resource constraints and task needs.

Abstract

Graph learning has attracted significant attention due to its widespread real-world applications. Current mainstream approaches rely on text node features and obtain initial node embeddings through shallow embedding learning using GNNs, which shows limitations in capturing deep textual semantics. Recent advances in Large Language Models (LLMs) have demonstrated superior capabilities in understanding text semantics, transforming traditional text feature processing. This paper proposes a novel framework that combines Graph Transformer architecture with LLM-enhanced node features. Specifically, we leverage LLMs to generate rich semantic representations of text nodes, which are then processed by a multi-head self-attention mechanism in the Graph Transformer to capture both local and global graph structural information. Our model utilizes the Transformer's attention mechanism to dynamically aggregate neighborhood information while preserving the semantic richness provided by LLM embeddings. Experimental results demonstrate that the LLM-enhanced node features significantly improve the performance of graph learning models on node classification tasks. This approach shows promising results across multiple graph learning tasks, offering a practical direction for combining graph networks with language models.