Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Boosting Graph Neural Network Expressivity with Learnable Lanczos Constraints

Niloofar Azizi, Nils Kriege, Horst Bischof

TL;DR

The paper tackles the limited expressivity of standard GNNs, which are bounded by $1$-WL and struggle with link prediction due to automorphic nodes. It proposes LLwLC, a Learnable Lanczos framework that injects linear constraints—derived from induced subgraphs—into the Laplacian eigenbasis to learn richer spectral features, implemented via a two-loop Lanczos process and a constrained LS solver. Two constraint policies are explored: Neumann eigenvalue constraints for linking representations and vertex-deleted subgraph constraints to distinguish graphs with identical WL signatures; stochastic constraint selection keeps computation practical. Empirically, LLwLCNet achieves state-of-the-art or competitive LP performance on benchmarks with far fewer parameters and data, and demonstrates substantial speedups, validating its theoretical claims on enhanced expressivity and practical utility.

Abstract

Graph Neural Networks (GNNs) excel in handling graph-structured data but often underperform in link prediction tasks compared to classical methods, mainly due to the limitations of the commonly used message-passing principle. Notably, their ability to distinguish non-isomorphic graphs is limited by the 1-dimensional Weisfeiler-Lehman test. Our study presents a novel method to enhance the expressivity of GNNs by embedding induced subgraphs into the graph Laplacian matrix's eigenbasis. We introduce a Learnable Lanczos algorithm with Linear Constraints (LLwLC), proposing two novel subgraph extraction strategies: encoding vertex-deleted subgraphs and applying Neumann eigenvalue constraints. For the former, we demonstrate the ability to distinguish graphs that are indistinguishable by 2-WL, while maintaining efficient time complexity. The latter focuses on link representations enabling differentiation between $k$-regular graphs and node automorphism, a vital aspect for link prediction tasks. Our approach results in an extremely lightweight architecture, reducing the need for extensive training datasets. Empirically, our method improves performance in challenging link prediction tasks across benchmark datasets, establishing its practical utility and supporting our theoretical findings. Notably, LLwLC achieves 20x and 10x speedup by only requiring 5% and 10% data from the PubMed and OGBL-Vessel datasets while comparing to the state-of-the-art.

Boosting Graph Neural Network Expressivity with Learnable Lanczos Constraints

TL;DR

The paper tackles the limited expressivity of standard GNNs, which are bounded by -WL and struggle with link prediction due to automorphic nodes. It proposes LLwLC, a Learnable Lanczos framework that injects linear constraints—derived from induced subgraphs—into the Laplacian eigenbasis to learn richer spectral features, implemented via a two-loop Lanczos process and a constrained LS solver. Two constraint policies are explored: Neumann eigenvalue constraints for linking representations and vertex-deleted subgraph constraints to distinguish graphs with identical WL signatures; stochastic constraint selection keeps computation practical. Empirically, LLwLCNet achieves state-of-the-art or competitive LP performance on benchmarks with far fewer parameters and data, and demonstrates substantial speedups, validating its theoretical claims on enhanced expressivity and practical utility.

Abstract

Graph Neural Networks (GNNs) excel in handling graph-structured data but often underperform in link prediction tasks compared to classical methods, mainly due to the limitations of the commonly used message-passing principle. Notably, their ability to distinguish non-isomorphic graphs is limited by the 1-dimensional Weisfeiler-Lehman test. Our study presents a novel method to enhance the expressivity of GNNs by embedding induced subgraphs into the graph Laplacian matrix's eigenbasis. We introduce a Learnable Lanczos algorithm with Linear Constraints (LLwLC), proposing two novel subgraph extraction strategies: encoding vertex-deleted subgraphs and applying Neumann eigenvalue constraints. For the former, we demonstrate the ability to distinguish graphs that are indistinguishable by 2-WL, while maintaining efficient time complexity. The latter focuses on link representations enabling differentiation between -regular graphs and node automorphism, a vital aspect for link prediction tasks. Our approach results in an extremely lightweight architecture, reducing the need for extensive training datasets. Empirically, our method improves performance in challenging link prediction tasks across benchmark datasets, establishing its practical utility and supporting our theoretical findings. Notably, LLwLC achieves 20x and 10x speedup by only requiring 5% and 10% data from the PubMed and OGBL-Vessel datasets while comparing to the state-of-the-art.
Paper Structure (31 sections, 6 theorems, 11 equations, 2 figures, 5 tables, 3 algorithms)

This paper contains 31 sections, 6 theorems, 11 equations, 2 figures, 5 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Notations
  4. Spectral Graph Convolutional Networks
  5. Lanczos with Linear Constraint Networks
  6. Lanczos Algorithm with Linear Constraints
  7. LLwLCNet
  8. Constraint Matrix $\mathbf{C}$
  9. Full Block
  10. Full Architecture
  11. Lanczos Algorithm with Linear Constraint Convergence
  12. Perturbation and Error Study
  13. Greenbaum's Results greenbaum1989behavior
  14. Subgraph Extraction Policy
  15. Neumann Eigenvalue
  16. ...and 16 more sections

Key Result

Theorem 1

Let $\mathcal{U}$ and $\tilde{\mathcal{U}}$ be the eigenspaces corresponding to the smallest eigenvalues $\lambda$ and $\tilde{\lambda}$ of the symmetric matrices $\mathbf{L}$ and $\tilde{\mathbf{L}} = \mathbf{L} + \mathbf{E}$, respectively. Then for any $\bm{u} \in \mathcal{U}$ and $\tilde{\bm{u}}

Figures (2)

  • Figure 1: The comprehensive workflow of LLwLC: We introduce a novel eigenbasis to boost the expressivity of GNNs by introducing constraints. For the two $2$-regular input graphs (first dotted block), the constraint matrix $\mathbf{C}$ (two columns shown in the second dotted block) is derived. Specifically, as depicted, for the first Neumann eigenvalue constraint (focused on link representation), we integrate the degrees of nodes into one column of the constraint matrix $\bm{C}$ such that $\mathbf{C}{^\top}\bm{f}=0$ ensures $\bm{f}(x) - \bm{f}(y) = 0$ for adjacent nodes $x$ and $y$ in the boundary. To predict the dotted link, nodes up to two hops away are analyzed: one-hop in light blue and two-hop in dark blue, centered on yellow query nodes. Based on the input graph Laplacian matrix $\mathbf{L}$ and the constraint matrix $\mathbf{C}$, we address the eigenvalue problem under these linear constraints using the Lanczos algorithm with the linear constraint consisting of an outer loop (Lanczos algorithm) and an inner loop ($\mathbf{P}(\nu)$ to $\mathcal{N}(\mathbf{C})$ with LSQR algorithm) resulting in the eigenpairs $\mathbf{V} = \mathbf{Q}\mathbf{B}$ and $\mathbf{R}$. The third block demonstrates constructing eigenbases for the two 2-regular graphs, ensuring features conform to constraints (indicated in blue). We stack multiple blocks, where block $i$ learns the new features $\mathbf{X}_{i+1}=\sigma(\mathbf{V}f_i(\mathbf{R})\mathbf{V}{^\top}\mathbf{X}_i\mathbf{W})$ by applying a multi-layer perceptron $f_i$ to the eigenvalue matrix $\mathbf{R}$. This sequence concludes with global pooling and a fully connected layer to output the predicted link probability $\hat{y}$.
  • Figure 2: The $C_6$ graph shows three different orbits of node pairs with three different colors. Both BUDDY chamberlain2022 features and our proposed features can distinguish them.

Theorems & Definitions (10)

  • Theorem 1
  • Theorem 2
  • Example 1
  • Proposition 1
  • Example 2
  • Theorem 3
  • Proposition 2
  • proof
  • Proposition 3
  • proof