Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

There is more to graphs than meets the eye: Learning universal features with self-supervision

Laya Das, Sai Munikoti, Nrushad Joshi, Mahantesh Halappanavar

TL;DR

This work addresses the limitation of single-graph self-supervised learning by proposing Universal Self-Supervised Learning (U-SSL), which learns representations that generalize across multiple graphs within the same family. The framework combines graph-specific encoders that homogenize disparate node features to a common space with a universal representation learning module that extracts generalizable features end-to-end, minimizing a joint SSL objective $    \mathcal{L}_{USSL}= \sum_{i=1}^N \mathcal{L}_{SSL,i}(\mathbf{X_i},\mathbf{A_i};\mathbf{\Theta_i},\mathbf{\Phi},\mathbf{\Gamma})$. Empirical results on six citation networks and diverse graph families show that U-SSL yields modest but consistent gains in node classification accuracy over SSL, improves efficiency (about 6% faster per epoch), and scales to larger graphs like OGBN-arxiv with up to ~8% improvement over SSL. The framework also demonstrates adaptability to unseen graphs by retraining only a graph-specific encoder for a new dataset, and can accommodate multiple pretext tasks to further boost performance. Overall, U-SSL advances toward foundation graph models by enabling end-to-end learning from multiple graphs, with promising implications for cross-graph transfer and robust, reusable graph representations.

Abstract

We study the problem of learning features through self-supervision that are generalisable to multiple graphs. State-of-the-art graph self-supervision restricts training to only one graph, resulting in graph-specific models that are incompatible with different but related graphs. We hypothesize that training with more than one graph that belong to the same family can improve the quality of the learnt representations. However, learning universal features from disparate node/edge features in different graphs is non-trivial. To address this challenge, we first homogenise the disparate features with graph-specific encoders that transform the features into a common space. A universal representation learning module then learns generalisable features on this common space. We show that compared to traditional self-supervision with one graph, our approach results in (1) better performance on downstream node classification, (2) learning features that can be re-used for unseen graphs of the same family, (3) more efficient training and (4) compact yet generalisable models. We also show ability of the proposed framework to deliver these benefits for relatively larger graphs. In this paper, we present a principled way to design foundation graph models that learn from more than one graph in an end-to-end manner, while bridging the gap between self-supervised and supervised performance.

There is more to graphs than meets the eye: Learning universal features with self-supervision

TL;DR

This work addresses the limitation of single-graph self-supervised learning by proposing Universal Self-Supervised Learning (U-SSL), which learns representations that generalize across multiple graphs within the same family. The framework combines graph-specific encoders that homogenize disparate node features to a common space with a universal representation learning module that extracts generalizable features end-to-end, minimizing a joint SSL objective . Empirical results on six citation networks and diverse graph families show that U-SSL yields modest but consistent gains in node classification accuracy over SSL, improves efficiency (about 6% faster per epoch), and scales to larger graphs like OGBN-arxiv with up to ~8% improvement over SSL. The framework also demonstrates adaptability to unseen graphs by retraining only a graph-specific encoder for a new dataset, and can accommodate multiple pretext tasks to further boost performance. Overall, U-SSL advances toward foundation graph models by enabling end-to-end learning from multiple graphs, with promising implications for cross-graph transfer and robust, reusable graph representations.

Abstract

We study the problem of learning features through self-supervision that are generalisable to multiple graphs. State-of-the-art graph self-supervision restricts training to only one graph, resulting in graph-specific models that are incompatible with different but related graphs. We hypothesize that training with more than one graph that belong to the same family can improve the quality of the learnt representations. However, learning universal features from disparate node/edge features in different graphs is non-trivial. To address this challenge, we first homogenise the disparate features with graph-specific encoders that transform the features into a common space. A universal representation learning module then learns generalisable features on this common space. We show that compared to traditional self-supervision with one graph, our approach results in (1) better performance on downstream node classification, (2) learning features that can be re-used for unseen graphs of the same family, (3) more efficient training and (4) compact yet generalisable models. We also show ability of the proposed framework to deliver these benefits for relatively larger graphs. In this paper, we present a principled way to design foundation graph models that learn from more than one graph in an end-to-end manner, while bridging the gap between self-supervised and supervised performance.
Paper Structure (27 sections, 6 equations, 1 figure, 8 tables)

This paper contains 27 sections, 6 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Graph neural networks and graph transformers
  4. Graph representation learning with self-supervision
  5. Learning Universal Features with Graph self-supervision
  6. Problem Formulation and Hypothesis
  7. Graph-specific Encoder
  8. Universal Representation Learning Module
  9. Pre-training and fine-tuning U-SSL models
  10. Experiments
  11. Results
  12. There is more to graphs than meets the eye
  13. Efficacy:
  14. Efficiency:
  15. Scalability:
  16. ...and 12 more sections

Figures (1)

  • Figure 1: Universal Self-supervised Learning (U-SSL) across graphs. (a) Model architecture for U-SSL with graph-specific ($\mathbf{\Theta_i}$) and universal ($\mathbf{\Phi}$) parameters. (b) U-SSL pre-training with two graphs, $\mathcal{G}_1$ and $\mathcal{G}_2$. (c) Downstream task learning for individual graphs. Hatched boxes represent frozen parameters ($\mathbf{\Theta_i,\Phi})$, and shaded boxes represent learnable parameters ($\mathbf{\Psi_i}$).

Theorems & Definitions (2)

  • Definition 1
  • Definition 2