Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhance Graph Alignment for Large Language Models

Haitong Luo, Xuying Meng, Suhang Wang, Tianxiang Zhao, Fali Wang, Hanyun Cao, Yujun Zhang

TL;DR

Graph Alignment Large Language Models (GALLM) is proposed to benefit from aligned task templates and Experimental evaluations on four datasets demonstrate substantial improvements in supervised learning, multi-dataset generalizability, and particularly in zero-shot capability, highlighting the model's potential as a graph foundation model.

Abstract

Graph-structured data is prevalent in the real world. Recently, due to the powerful emergent capabilities, Large Language Models (LLMs) have shown promising performance in modeling graphs. The key to effectively applying LLMs on graphs is converting graph data into a format LLMs can comprehend. Graph-to-token approaches are popular in enabling LLMs to process graph information. They transform graphs into sequences of tokens and align them with text tokens through instruction tuning, where self-supervised instruction tuning helps LLMs acquire general knowledge about graphs, and supervised fine-tuning specializes LLMs for the downstream tasks on graphs. Despite their initial success, we find that existing methods have a misalignment between self-supervised tasks and supervised downstream tasks, resulting in negative transfer from self-supervised fine-tuning to downstream tasks. To address these issues, we propose Graph Alignment Large Language Models (GALLM) to benefit from aligned task templates. In the self-supervised tuning stage, we introduce a novel text matching task using templates aligned with downstream tasks. In the task-specific tuning stage, we propose two category prompt methods that learn supervision information from additional explanation with further aligned templates. Experimental evaluations on four datasets demonstrate substantial improvements in supervised learning, multi-dataset generalizability, and particularly in zero-shot capability, highlighting the model's potential as a graph foundation model.

Enhance Graph Alignment for Large Language Models

TL;DR

Graph Alignment Large Language Models (GALLM) is proposed to benefit from aligned task templates and Experimental evaluations on four datasets demonstrate substantial improvements in supervised learning, multi-dataset generalizability, and particularly in zero-shot capability, highlighting the model's potential as a graph foundation model.

Abstract

Graph-structured data is prevalent in the real world. Recently, due to the powerful emergent capabilities, Large Language Models (LLMs) have shown promising performance in modeling graphs. The key to effectively applying LLMs on graphs is converting graph data into a format LLMs can comprehend. Graph-to-token approaches are popular in enabling LLMs to process graph information. They transform graphs into sequences of tokens and align them with text tokens through instruction tuning, where self-supervised instruction tuning helps LLMs acquire general knowledge about graphs, and supervised fine-tuning specializes LLMs for the downstream tasks on graphs. Despite their initial success, we find that existing methods have a misalignment between self-supervised tasks and supervised downstream tasks, resulting in negative transfer from self-supervised fine-tuning to downstream tasks. To address these issues, we propose Graph Alignment Large Language Models (GALLM) to benefit from aligned task templates. In the self-supervised tuning stage, we introduce a novel text matching task using templates aligned with downstream tasks. In the task-specific tuning stage, we propose two category prompt methods that learn supervision information from additional explanation with further aligned templates. Experimental evaluations on four datasets demonstrate substantial improvements in supervised learning, multi-dataset generalizability, and particularly in zero-shot capability, highlighting the model's potential as a graph foundation model.

Paper Structure

This paper contains 34 sections, 8 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Learning on Graphs
  4. Large Language Models for Graphs
  5. Methodology
  6. Notations
  7. Instruction Tuning Pipeline Design
  8. Self-Supervised Instruction Tuning
  9. Self-Supervised Task Design
  10. Mixing Negative Sampling Strategy.
  11. Task-specific Instruction Tuning
  12. Category-Enhanced Manual Prompt Tuning.
  13. Category-Enhanced Soft Prompt Tuning.
  14. EXPERIMENTS
  15. Experiment Setup
  16. ...and 19 more sections

Figures (4)

  • Figure 1: Preliminary study on the negative transfer of existing self-supervised tasks.
  • Figure 2: The overall architecture of GALLM with two-stage instruction tuning paradigm.
  • Figure 3: Illustration of category-enhanced soft prompt tuning. We show only the category prompt, keeping other parts (e.g., graph data input format) unchanged. In this tuning, the parameters of virtual token embeddings, soft prompt encoder, and graph projector are tuned.
  • Figure 4: Compatibility evaluation with different graph description templates and base LLM models.