Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Graph Foundation Models: Learning Generalities Across Graphs via Task-Trees

Zehong Wang, Zheyuan Zhang, Tianyi Ma, Nitesh V Chawla, Chuxu Zhang, Yanfang Ye

TL;DR

This work tackles cross-task generalization in graphs by introducing task-trees, a unified learning unit that aligns node-, edge-, and graph-level tasks. The authors provide a theoretical foundation—stability, transferability, and generalization bounds—for pretraining on task-trees and demonstrate a graph foundation model, GIT, pretrained on diverse task-trees to generalize across 32 graphs in five domains. They further show that domain specialization via instruction tuning can match or surpass domain-specific experts, while maintaining broad generalization capabilities. Empirically, GIT-G and GIT-S excel under fine-tuning, in-context learning, and zero-shot settings, and task-trees prove more efficient and effective than subgraphs for cross-domain learning. The work suggests a scalable, principled route to general-purpose graph reasoning with practical benefits for real-world heterogeneous graph data.

Abstract

Foundation models are pretrained on large-scale corpora to learn generalizable patterns across domains and tasks -- such as contours, textures, and edges in images, or tokens and sentences in text. In contrast, discovering such generalities in graph-structured data, especially across heterogeneous graph tasks, remains an open challenge. To address this, we propose a novel approach to cross-task generalization in graphs via task-trees, which serve as unified learning instances aligning node-, edge-, and graph-level tasks. We theoretically analyze the stability, transferability, and generalization properties of task-trees, showing that pretraining a graph neural network (GNN) on diverse task-trees with a reconstruction objective induces transferable knowledge. This enables efficient adaptation to downstream tasks with minimal fine-tuning. To validate our framework, we introduce Graph Generality Identifier on Task-Trees (GIT), a graph foundation model that demonstrates strong performance on over 30 graphs across five domains via fine-tuning, in-context learning, and zero-shot generalization. Code and data are available at https://github.com/Zehong-Wang/GIT.

Towards Graph Foundation Models: Learning Generalities Across Graphs via Task-Trees

TL;DR

This work tackles cross-task generalization in graphs by introducing task-trees, a unified learning unit that aligns node-, edge-, and graph-level tasks. The authors provide a theoretical foundation—stability, transferability, and generalization bounds—for pretraining on task-trees and demonstrate a graph foundation model, GIT, pretrained on diverse task-trees to generalize across 32 graphs in five domains. They further show that domain specialization via instruction tuning can match or surpass domain-specific experts, while maintaining broad generalization capabilities. Empirically, GIT-G and GIT-S excel under fine-tuning, in-context learning, and zero-shot settings, and task-trees prove more efficient and effective than subgraphs for cross-domain learning. The work suggests a scalable, principled route to general-purpose graph reasoning with practical benefits for real-world heterogeneous graph data.

Abstract

Foundation models are pretrained on large-scale corpora to learn generalizable patterns across domains and tasks -- such as contours, textures, and edges in images, or tokens and sentences in text. In contrast, discovering such generalities in graph-structured data, especially across heterogeneous graph tasks, remains an open challenge. To address this, we propose a novel approach to cross-task generalization in graphs via task-trees, which serve as unified learning instances aligning node-, edge-, and graph-level tasks. We theoretically analyze the stability, transferability, and generalization properties of task-trees, showing that pretraining a graph neural network (GNN) on diverse task-trees with a reconstruction objective induces transferable knowledge. This enables efficient adaptation to downstream tasks with minimal fine-tuning. To validate our framework, we introduce Graph Generality Identifier on Task-Trees (GIT), a graph foundation model that demonstrates strong performance on over 30 graphs across five domains via fine-tuning, in-context learning, and zero-shot generalization. Code and data are available at https://github.com/Zehong-Wang/GIT.

Paper Structure

This paper contains 53 sections, 5 theorems, 46 equations, 10 figures, 28 tables.

Table of Contents

  1. Introduction
  2. Task-Trees: Rethinking Basic Learning Instances on Graphs
  3. Preliminary
  4. Task-Tree Construction and Encoding
  5. Comparison to Existing Works
  6. Theoretical Analysis of Task-Trees
  7. Graph Generality Identifier on Task-Trees
  8. GIT-G: Pretraining to Acquire General Knowledge
  9. GIT-S: Specification via Instruction Tuning
  10. Experiment
  11. Experimental Setup
  12. Main Results
  13. Comparison to State-of-The-Art Methods
  14. Ablation on Training Strategies
  15. Comparison to Domain Experts
  16. ...and 38 more sections

Key Result

Theorem 3.1

Given two $L$-layer task-trees $T_t^1$ and $T_t^2$, with task-relevant nodes $\{v_1, ..., v_n\}$ and $\{v_1, ..., v_m\}$, respectively. The distance between task-trees is defined as $\Delta := \| \phi(T_1^t) - \phi(T_2^t) \|$ with where $\phi$ is the GNN encoder, $T_i$ is the computation tree corresponding to node $i$, and ${\mathcal{C}}_1, {\mathcal{C}}_2$ are constants related to the encoder, a

Figures (10)

  • Figure 1: The formulation of task-trees.
  • Figure 2: subtree information examples.
  • Figure 3: The model performance on all datasets in the fine-tuning setting. For GIT, the best result is selected between GIT-G and GIT-S, while the best baseline performance is chosen among GCN/GAT/GIN, BGRL, GraphMAE, and OFA.
  • Figure 4: Training efficiency between task-tree and subgraph versions of GIT.
  • Figure 5: The domain regularizer controls the distance between datasets while preserving the structure within each of them.
  • ...and 5 more figures

Theorems & Definitions (12)

  • Definition 2.1: Task-Relevant Nodes
  • Definition 2.2: Computation Trees chuang2022tree
  • Definition 2.3: Task-Trees
  • Theorem 3.1: Stability on Task-Trees
  • Theorem 3.3: Transferability on Task-Trees
  • Theorem 3.5: Generalization on Task-Trees
  • proof
  • Lemma 4.1
  • proof
  • proof
  • ...and 2 more