Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniques

TL;DR

The paper addresses the lack of a comprehensive technical review of integrating Large Language Models with Graph Representation Learning. It introduces a two-component taxonomy—knowledge extractors and knowledge organizers—and two operation techniques—integration and training strategies—to structure the field. It surveys existing methods across tasks like attribute, structure, and label extraction; and organizer designs (GNN-centric, LLM-centric, GNN+LLM), with various integration and training strategies. The discussion highlights future directions and practical implications for building graph foundation models that leverage LLMs.

Abstract

The integration of Large Language Models (LLMs) with Graph Representation Learning (GRL) marks a significant evolution in analyzing complex data structures. This collaboration harnesses the sophisticated linguistic capabilities of LLMs to improve the contextual understanding and adaptability of graph models, thereby broadening the scope and potential of GRL. Despite a growing body of research dedicated to integrating LLMs into the graph domain, a comprehensive review that deeply analyzes the core components and operations within these models is notably lacking. Our survey fills this gap by proposing a novel taxonomy that breaks down these models into primary components and operation techniques from a novel technical perspective. We further dissect recent literature into two primary components including knowledge extractors and organizers, and two operation techniques including integration and training stratigies, shedding light on effective model design and training strategies. Additionally, we identify and explore potential future research avenues in this nascent yet underexplored field, proposing paths for continued progress.

Figures (4)

• Figure 1: The techniques of GRL with LLMs.
• Figure 2: The illustration of graph knowledge extractors on attribute, structure and label information with LLMs.
• Figure 3: The illustration of different knowledge organizers: GNN-centric, LLM-centric, and GNN+LLM organizers.
• Figure 4: The illustration of different knowledge integration strategies.