GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuning

TL;DR

This work tackles the instability of few-shot fine-tuning for Graph Foundation Models under multi-domain pre-training. It proposes GRAVER, a framework that builds transferable graph vocabularies through ego-graph disentanglement, models them with graphon-based experts for structure and feature tokens, and uses a hierarchical MoE-CoE routing to assemble source-domain knowledge for in-context augmentation. The approach is supported by theoretical results on vocabulary transferability and convergence, and validated by extensive experiments across seven multi-domain datasets, showing improvements in effectiveness, robustness, and efficiency over 15 baselines for both node and graph classification. Overall, GRAVER advances robust cross-domain graph learning by enabling stable knowledge transfer with limited labeled data, with practical implications for real-world graph applications.

Abstract

Inspired by the remarkable success of foundation models in language and vision, Graph Foundation Models (GFMs) hold significant promise for broad applicability across diverse graph tasks and domains. However, existing GFMs struggle with unstable few-shot fine-tuning, where both performance and adaptation efficiency exhibit significant fluctuations caused by the randomness in the support sample selection and structural discrepancies between the pre-trained and target graphs. How to fine-tune GFMs robustly and efficiently to enable trustworthy knowledge transfer across domains and tasks is the major challenge. In this paper, we propose GRAVER, a novel Generative gRAph VocabulariEs for Robust GFM fine-tuning framework that tackles the aforementioned instability via generative augmentations. Specifically, to identify transferable units, we analyze and extract key class-specific subgraph patterns by ego-graph disentanglement and validate their transferability both theoretically and empirically. To enable effective pre-training across diverse domains, we leverage a universal task template based on ego-graph similarity and construct graph vocabularies via graphon-based generative experts. To facilitate robust and efficient prompt fine-tuning, we grave the support samples with in-context vocabularies, where the lightweight MoE-CoE network attentively routes knowledge from source domains. Extensive experiments demonstrate the superiority of GRAVER over effectiveness, robustness, and efficiency on downstream few-shot node and graph classification tasks compared with 15 state-of-the-art baselines.

Figures (16)

• Figure 1: Case study of the fine-tuning instability. When support samples share structural patterns with pre-trained graphs (e.g., triangle), fine-tuning is efficient and stable (Good Case). In contrast, mismatched patterns (e.g., ladder) lead to poor convergence and lower accuracy (Bad Case). Loss and accuracy curves illustrate that random support selection causes high performance deviation, highlighting the need for structure-aware augmentation.
• Figure 2: Framework of Graver. (1) aligns semantics via LLM-enhanced raw text, and disentangles ego-graph into transferable patterns. (2) models structure-feature tokens with graphon experts. (3) assembles class-aware vocabularies to augment support samples for robust cross-domain fine-tuning.
• Figure 3: $m$-shot node classification.
• Figure 4: 5-shot graph classification (Wiki-CS).
• Figure 5: Ablation studies (ogbn-Home).

Theorems & Definitions (8)

• Proposition 1: Vocabulary Transferability
• Proposition 2: Generation Distributional Convergence
• proof
• proof
• proof
• Lemma 1: Consistency of Vocabulary Graphon Estimators
• proof
• proof