Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Instruction-based Hypergraph Pretraining

Mingdai Yang, Zhiwei Liu, Liangwei Yang, Xiaolong Liu, Chen Wang, Hao Peng, Philip S. Yu

TL;DR

This paper proposes a novel pretraining framework named Instruction-based Hypergraph Pretraining, inspired by instruction-based prompts widely used in pre-trained language models, and introduces instructions into graph pertaining to overcome the discrepancy between pretraining and downstream tasks.

Abstract

Pretraining has been widely explored to augment the adaptability of graph learning models to transfer knowledge from large datasets to a downstream task, such as link prediction or classification. However, the gap between training objectives and the discrepancy between data distributions in pretraining and downstream tasks hinders the transfer of the pretrained knowledge. Inspired by instruction-based prompts widely used in pretrained language models, we introduce instructions into graph pretraining. In this paper, we propose a novel pretraining framework named Instruction-based Hypergraph Pretraining. To overcome the discrepancy between pretraining and downstream tasks, text-based instructions are applied to provide explicit guidance on specific tasks for representation learning. Compared to learnable prompts, whose effectiveness depends on the quality and the diversity of training data, text-based instructions intrinsically encapsulate task information and support the model to generalize beyond the structure seen during pretraining. To capture high-order relations with task information in a context-aware manner, a novel prompting hypergraph convolution layer is devised to integrate instructions into information propagation in hypergraphs. Extensive experiments conducted on three public datasets verify the superiority of IHP in various scenarios.

Instruction-based Hypergraph Pretraining

TL;DR

This paper proposes a novel pretraining framework named Instruction-based Hypergraph Pretraining, inspired by instruction-based prompts widely used in pre-trained language models, and introduces instructions into graph pertaining to overcome the discrepancy between pretraining and downstream tasks.

Abstract

Pretraining has been widely explored to augment the adaptability of graph learning models to transfer knowledge from large datasets to a downstream task, such as link prediction or classification. However, the gap between training objectives and the discrepancy between data distributions in pretraining and downstream tasks hinders the transfer of the pretrained knowledge. Inspired by instruction-based prompts widely used in pretrained language models, we introduce instructions into graph pretraining. In this paper, we propose a novel pretraining framework named Instruction-based Hypergraph Pretraining. To overcome the discrepancy between pretraining and downstream tasks, text-based instructions are applied to provide explicit guidance on specific tasks for representation learning. Compared to learnable prompts, whose effectiveness depends on the quality and the diversity of training data, text-based instructions intrinsically encapsulate task information and support the model to generalize beyond the structure seen during pretraining. To capture high-order relations with task information in a context-aware manner, a novel prompting hypergraph convolution layer is devised to integrate instructions into information propagation in hypergraphs. Extensive experiments conducted on three public datasets verify the superiority of IHP in various scenarios.
Paper Structure (37 sections, 7 equations, 7 figures, 6 tables)

This paper contains 37 sections, 7 equations, 7 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Hypergraph Learning
  4. Graph Pretraining and Prompting
  5. Graph Learning with Text Information
  6. Preliminaries
  7. Problem Formulation
  8. Hypergraph
  9. Method
  10. Embedding Layer
  11. Hypergraph Construction
  12. Prompt Hypergraph Convolution
  13. Hyperedge Initialization
  14. Information Prompting
  15. Node update
  16. ...and 22 more sections

Figures (7)

  • Figure 1: A toy example of how related instructions benefit pretext and downstream tasks with high-order relations.
  • Figure 2: (a) The overall framework of IHP. The target nodes are the nodes existing in both pretraining and finetuning stages, and other nodes are defined as context nodes; (b) The illustration of the Prompt Layer; (c) The illustration of the PHC Layer.
  • Figure 3: Performance of IHP w.r.t. different instruction constructions.
  • Figure 4: Performance of IHP w.r.t. random, frozen or updated target node embeddings during the finetuning stage.
  • Figure 5: IHP finetuned with pretrained data added if the learning rates of target and context node embeddings are the same, i.e., $\lambda_\text{t}=1$. The curve represents the performance, and the bars denote the ratios of the edges added from pretraining in finetuning.
  • ...and 2 more figures

Theorems & Definitions (1)

  • Definition 1