Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks

Zhili Wang, Shimin Di, Lei Chen, Xiaofang Zhou

TL;DR

S2PGNN first carefully summarizes a search space of fine-tuning strategies that is suitable for GNNs, which is expressive enough to enable powerful strategies to be searched and integrates an efficient search algorithm to solve the computationally expensive search problem from a discrete and large space.

Abstract

Recently, graph neural networks (GNNs) have shown its unprecedented success in many graph-related tasks. However, GNNs face the label scarcity issue as other neural networks do. Thus, recent efforts try to pre-train GNNs on a large-scale unlabeled graph and adapt the knowledge from the unlabeled graph to the target downstream task. The adaptation is generally achieved by fine-tuning the pre-trained GNNs with a limited number of labeled data. Despite the importance of fine-tuning, current GNNs pre-training works often ignore designing a good fine-tuning strategy to better leverage transferred knowledge and improve the performance on downstream tasks. Only few works start to investigate a better fine-tuning strategy for pre-trained GNNs. But their designs either have strong assumptions or overlook the data-aware issue for various downstream datasets. Therefore, we aim to design a better fine-tuning strategy for pre-trained GNNs to improve the model performance in this paper. Given a pre-trained GNN, we propose to search to fine-tune pre-trained graph neural networks for graph-level tasks (S2PGNN), which adaptively design a suitable fine-tuning framework for the given labeled data on the downstream task. To ensure the improvement brought by searching fine-tuning strategy, we carefully summarize a proper search space of fine-tuning framework that is suitable for GNNs. The empirical studies show that S2PGNN can be implemented on the top of 10 famous pre-trained GNNs and consistently improve their performance. Besides, S2PGNN achieves better performance than existing fine-tuning strategies within and outside the GNN area. Our code is publicly available at \url{https://anonymous.4open.science/r/code_icde2024-A9CB/}.

Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks

TL;DR

S2PGNN first carefully summarizes a search space of fine-tuning strategies that is suitable for GNNs, which is expressive enough to enable powerful strategies to be searched and integrates an efficient search algorithm to solve the computationally expensive search problem from a discrete and large space.

Abstract

Recently, graph neural networks (GNNs) have shown its unprecedented success in many graph-related tasks. However, GNNs face the label scarcity issue as other neural networks do. Thus, recent efforts try to pre-train GNNs on a large-scale unlabeled graph and adapt the knowledge from the unlabeled graph to the target downstream task. The adaptation is generally achieved by fine-tuning the pre-trained GNNs with a limited number of labeled data. Despite the importance of fine-tuning, current GNNs pre-training works often ignore designing a good fine-tuning strategy to better leverage transferred knowledge and improve the performance on downstream tasks. Only few works start to investigate a better fine-tuning strategy for pre-trained GNNs. But their designs either have strong assumptions or overlook the data-aware issue for various downstream datasets. Therefore, we aim to design a better fine-tuning strategy for pre-trained GNNs to improve the model performance in this paper. Given a pre-trained GNN, we propose to search to fine-tune pre-trained graph neural networks for graph-level tasks (S2PGNN), which adaptively design a suitable fine-tuning framework for the given labeled data on the downstream task. To ensure the improvement brought by searching fine-tuning strategy, we carefully summarize a proper search space of fine-tuning framework that is suitable for GNNs. The empirical studies show that S2PGNN can be implemented on the top of 10 famous pre-trained GNNs and consistently improve their performance. Besides, S2PGNN achieves better performance than existing fine-tuning strategies within and outside the GNN area. Our code is publicly available at \url{https://anonymous.4open.science/r/code_icde2024-A9CB/}.
Paper Structure (30 sections, 13 equations, 4 figures, 11 tables)

This paper contains 30 sections, 13 equations, 4 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph Neural Networks (GNNs)
  4. Manual GNNs
  5. Automated GNNs
  6. Fine-tuning Pre-trained GNNs
  7. Methodology
  8. Problem Formulation
  9. GNN Fine-tuning Search Space
  10. Backbone Convolution $\bm{\phi}_{conv}(\cdot)$
  11. Identity Augmentation $\bm{\phi}_{id}(\cdot)$
  12. Multi-scale Fusion $\bm{\phi}_{fuse}(\cdot)$
  13. Graph-level Readout $\bm{\phi}_{read}(\cdot)$
  14. GNN Fine-tuning Search Algorithm
  15. Update Fine-tuning Strategy by Re-parameterization
  16. ...and 15 more sections

Figures (4)

  • Figure 1: The illustration of overall GNN pre-training and fine-tuning framework.
  • Figure 2: Illustration of GNN fine-tuning strategies (refer to Fig. \ref{['fig: ours']} for the legend).
  • Figure 3: Illustration to the framework of S2PGNN built on top of pre-trained 5-layer GIN. The orange part indicates the search dimensions in S2PGNN. PT., FT., and Id.Aug. are abbreviations for pre-training, fine-tuning, and identity augmentation.
  • Figure 4: Illustration to other fine-tuning strategies (refer to Fig. \ref{['fig: ours']} for the legend).

Theorems & Definitions (4)

  • Remark 1: Prompt Tuning
  • Remark 2: Fine-tuning Technique Outside GNNs
  • Definition 1: The Fine-tuning Strategy Search Problem for Pre-trained GNNs
  • Remark 3: Space Complexity of $\bm{\Phi}_{ft}$