Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings

Duo Wang, Yuan Zuo, Fengzhi Li, Junjie Wu

TL;DR

TEA-GLM introduces a cross-dataset, cross-task zero-shot framework by aligning GNN representations with LLM token embeddings and mapping graph features into a fixed set of graph tokens via a linear projector. It jointly trains two contrastive objectives—instance-wise across graph views and feature-wise to LLM embeddings—enhanced by PCA-based alignment and a designed instruction paradigm that unifies node-, edge-, and graph-level tasks. Empirical results show TEA-GLM achieving state-of-the-art zero-shot performance on unseen datasets and tasks, with ablations confirming the critical roles of graph token embeddings and feature-wise alignment. This approach enables transferable graph reasoning with a fixed LLM, reducing task-specific fine-tuning while leveraging LLM knowledge for practical zero-shot graph learning.

Abstract

Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors.

LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings

TL;DR

TEA-GLM introduces a cross-dataset, cross-task zero-shot framework by aligning GNN representations with LLM token embeddings and mapping graph features into a fixed set of graph tokens via a linear projector. It jointly trains two contrastive objectives—instance-wise across graph views and feature-wise to LLM embeddings—enhanced by PCA-based alignment and a designed instruction paradigm that unifies node-, edge-, and graph-level tasks. Empirical results show TEA-GLM achieving state-of-the-art zero-shot performance on unseen datasets and tasks, with ablations confirming the critical roles of graph token embeddings and feature-wise alignment. This approach enables transferable graph reasoning with a fixed LLM, reducing task-specific fine-tuning while leveraging LLM knowledge for practical zero-shot graph learning.

Abstract

Zero-shot graph machine learning, especially with graph neural networks (GNNs), has garnered significant interest due to the challenge of scarce labeled data. While methods like self-supervised learning and graph prompt learning have been extensively explored, they often rely on fine-tuning with task-specific labels, limiting their effectiveness in zero-shot scenarios. Inspired by the zero-shot capabilities of instruction-fine-tuned large language models (LLMs), we introduce a novel framework named Token Embedding-Aligned Graph Language Model (TEA-GLM) that leverages LLMs as cross-dataset and cross-task zero-shot learners for graph machine learning. Concretely, we pretrain a GNN, aligning its representations with token embeddings of an LLM. We then train a linear projector that transforms the GNN's representations into a fixed number of graph token embeddings without tuning the LLM. A unified instruction is designed for various graph tasks at different levels, such as node classification (node-level) and link prediction (edge-level). These design choices collectively enhance our method's effectiveness in zero-shot learning, setting it apart from existing methods. Experiments show that our graph token embeddings help the LLM predictor achieve state-of-the-art performance on unseen datasets and tasks compared to other methods using LLMs as predictors.
Paper Structure (39 sections, 9 equations, 5 figures, 7 tables)

This paper contains 39 sections, 9 equations, 5 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Notations
  4. Token embeddings-aligned graph self-supervised learning
  5. Instance-wise contrastive learning with structural information
  6. Feature-wise contrastive learning with token embeddings
  7. Alignment tuning
  8. Instructions design
  9. Graph information provision
  10. Task description
  11. Graph token embeddings
  12. Training and evaluation strategy
  13. Experimental results
  14. Experimental setup
  15. Datasets
  16. ...and 24 more sections

Figures (5)

  • Figure 1: Framework of TEA-GLM
  • Figure 2: Ablation study results ("Seen datasets" are used to train the GNN and linear projector, while "unseen datasets" are not. "Unseen task" means the model wasn't trained for link prediction.)
  • Figure 3: Impact of number of graph token embeddings (Macro F1)
  • Figure 4: Impact of number of principal components (Macro F1)
  • Figure 5: Instructions for node classification and link prediction