Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Graph Laplacian Eigenvector-based Pre-training Method for Graph Neural Networks

Howard Dai, Nyambura Njenga, Hiren Madhu, Siddharth Viswanath, Ryan Pellico, Ian Adelstein, Smita Krishnaswamy

TL;DR

This work addresses structure-based pre-training for graph neural networks, aiming to capture global graph structure without relying on excessively deep architectures. It introduces the Laplacian Eigenvector Learning Module (LELM), a pre-training framework where a graph-level MLP and diffusion-based node features enable a GNN to predict the $k$ lowest-frequency Laplacian eigenvectors. The training objective combines an energy loss and an eigenvector loss, with a QR-based orthogonality constraint to ensure a proper eigenbasis and correct ordering, while preserving sign and basis invariances. Empirically, LELM yields consistent improvements on molecular property prediction tasks and outperforms several alternative structure-based targets, underscoring its potential as a versatile component for graph foundation models.

Abstract

The development of self-supervised graph pre-training methods is a crucial ingredient in recent efforts to design robust graph foundation models (GFMs). Structure-based pre-training methods are under-explored yet crucial for downstream applications which rely on underlying graph structure. In addition, pre-training traditional message passing GNNs to capture global and regional structure is often challenging due to the risk of oversmoothing as network depth increases. We address these gaps by proposing the Laplacian Eigenvector Learning Module (LELM), a novel pre-training module for graph neural networks (GNNs) based on predicting the low-frequency eigenvectors of the graph Laplacian. Moreover, LELM introduces a novel architecture that overcomes oversmoothing, allowing the GNN model to learn long-range interdependencies. Empirically, we show that models pre-trained via our framework outperform baseline models on downstream molecular property prediction tasks.

A Graph Laplacian Eigenvector-based Pre-training Method for Graph Neural Networks

TL;DR

This work addresses structure-based pre-training for graph neural networks, aiming to capture global graph structure without relying on excessively deep architectures. It introduces the Laplacian Eigenvector Learning Module (LELM), a pre-training framework where a graph-level MLP and diffusion-based node features enable a GNN to predict the lowest-frequency Laplacian eigenvectors. The training objective combines an energy loss and an eigenvector loss, with a QR-based orthogonality constraint to ensure a proper eigenbasis and correct ordering, while preserving sign and basis invariances. Empirically, LELM yields consistent improvements on molecular property prediction tasks and outperforms several alternative structure-based targets, underscoring its potential as a versatile component for graph foundation models.

Abstract

The development of self-supervised graph pre-training methods is a crucial ingredient in recent efforts to design robust graph foundation models (GFMs). Structure-based pre-training methods are under-explored yet crucial for downstream applications which rely on underlying graph structure. In addition, pre-training traditional message passing GNNs to capture global and regional structure is often challenging due to the risk of oversmoothing as network depth increases. We address these gaps by proposing the Laplacian Eigenvector Learning Module (LELM), a novel pre-training module for graph neural networks (GNNs) based on predicting the low-frequency eigenvectors of the graph Laplacian. Moreover, LELM introduces a novel architecture that overcomes oversmoothing, allowing the GNN model to learn long-range interdependencies. Empirically, we show that models pre-trained via our framework outperform baseline models on downstream molecular property prediction tasks.

Paper Structure

This paper contains 28 sections, 4 theorems, 38 equations, 2 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. Method
  4. Node feature augmentation
  5. Graph-level MLP
  6. Eigenvector prediction
  7. Experimental Results
  8. Limitations and Future Work
  9. Related works: GNN pre-training methods
  10. Graph Laplacian eigenvector applications
  11. Complete algorithms
  12. Loss function
  13. Loss function alternative expressions
  14. Orthogonality
  15. Energy and eigenvector losses are sign and basis invariant
  16. ...and 13 more sections

Key Result

Lemma 1

For any $R_\Psi$ and a single eigenvector prediction $u \in \mathbb{R}^n$, we have:

Figures (2)

  • Figure 1: Overview of the LELM pre-training pipeline. Here, "Base GNN" and "Downstream Prediction Head" can be any user-defined model architecture.
  • Figure 2: A comparison of predicted eigenvectors (blue) with ground-truth eigenvectors (orange) for a single molecular graph with $n = 21$ nodes. These predictions were produced by the standard GIN model on a validation example from the ZINC dataset. Node indices are sorted in increasing order of $\psi_2(i)$, and the sign orientation of the predicted vectors $\hat{u}_i$ is chosen such that $\hat{u}_i \cdot \psi_i\geq 0$.

Theorems & Definitions (9)

  • Definition 1: Basis invariance
  • Lemma 1: Energy loss is basis invariant
  • proof
  • Lemma 2: Eigenvector loss is basis invariant
  • proof
  • Lemma 3: Energy loss is rotation invariant
  • proof
  • Lemma 4: Uniqueness up to co-spectrality
  • proof