Towards Graph Foundation Models: A Study on the Generalization of Positional and Structural Encodings

TL;DR

The paper investigates whether learnable graph positional and structural encodings (PSEs), exemplified by GPSE, can serve as universal building blocks for graph foundation models. It demonstrates that GPSE can function as a universal node encoder under mild assumptions, yet downstream universality is not guaranteed without randomness or task-specific pretraining; it also shows GPSE can accelerate convergence and improve data efficiency in many settings, though performance is dataset-dependent. Through extensive experiments on synthetic expressivity benchmarks and real molecular datasets (e.g., ZINC-12k, MolNet), the study reveals that GPSE and its variants often outperform baselines in generalization and data-scarce regimes, while not universally surpassing all baselines across all tasks. The findings suggest PSEs hold significant potential as integral components of future graph foundation models, while underscoring the need for improved generalization mechanisms across diverse graph domains.

Abstract

Recent advances in integrating positional and structural encodings (PSEs) into graph neural networks (GNNs) have significantly enhanced their performance across various graph learning tasks. However, the general applicability of these encodings and their potential to serve as foundational representations for graphs remain uncertain. This paper investigates the fine-tuning efficiency, scalability with sample size, and generalization capability of learnable PSEs across diverse graph datasets. Specifically, we evaluate their potential as universal pre-trained models that can be easily adapted to new tasks with minimal fine-tuning and limited data. Furthermore, we assess the expressivity of the learned representations, particularly, when used to augment downstream GNNs. We demonstrate through extensive benchmarking and empirical analysis that PSEs generally enhance downstream models. However, some datasets may require specific PSE-augmentations to achieve optimal performance. Nevertheless, our findings highlight their significant potential to become integral components of future graph foundation models. We provide new insights into the strengths and limitations of PSEs, contributing to the broader discourse on foundation models in graph learning.

Figures (3)

• Figure 1: GPSE without RNF cannot differentiate graphs (a) and (b). An MPNN with the orbit partition as node features cannot differentiate graphs (c) and (d).
• Figure 2: GPSE variants outperform all other PSEs regardless of the amount of available training data on ZINC-12k. Notably, for less available training data, the advantage of GPSE and its variants is more strongly pronounced. However, for ToxCast the opposite is true. This figure shows downstream training with fractions of the ZINC-12k and ToxCast datasets. We show the difference in performance (MAE $\downarrow$ for ZINC-12K and AUROC $\uparrow$ for ToxCast) obtained by various PSEs, including GPSE$^{\boldsymbol{-}}$ and GPSE$^{\boldsymbol{+}}$. We show results on additional datasets in \ref{['figure:additionaldatasets_fewshot']}.
• Figure 3: In most cases, GPSE variants outperform other PSEs regardless of available training data on the MolNet datasets. There are, however, also examples where GPSE is outperformed by other PSEs for less available training data. Results on additional datasets beyond what is shown in \ref{['fig:zinc_fewshot']}. Downstream training with fractions of the MolNet datasets. We show the difference in performance (AUROC $\uparrow$) obtained by various PSEs, including GPSE$^{\boldsymbol{-}}$ and GPSE$^{\boldsymbol{+}}$.

Theorems & Definitions (5)

• Theorem 1
• Theorem 2
• proof
• Theorem 1
• proof