Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Foundations and Frontiers of Graph Learning Theory

Yu Huang, Min Zhou, Menglin Yang, Zhen Wang, Muhan Zhang, Jie Wang, Hong Xie, Hao Wang, Defu Lian, Enhong Chen

TL;DR

This paper surveys the theoretical foundations of graph learning, focusing on three core pillars—expressive power, generalization, and optimization—and also addresses long-range interactions via over-smoothing and over-squashing. It synthesizes WL-based expressivity, higher-order and subgraph approaches, invariance/equivariance, and connections to combinatorial problems, while outlining generalization bounds (VC-dim, Rademacher, PAC-Bayes, stability, GNTK) and optimization dynamics (NTK regime, initialization, normalization, sampling). The work also discusses practical strategies to mitigate deep-GNN pathologies (skip connections, ODE-based models, graph rewiring) and outlines open questions linking theory to real-world graph tasks. Overall, it provides a structured theory of graph learning with clear directions toward more powerful, generalizable, and scalable graph models, including graph transformers and geometry-aware architectures.

Abstract

Recent advancements in graph learning have revolutionized the way to understand and analyze data with complex structures. Notably, Graph Neural Networks (GNNs), i.e. neural network architectures designed for learning graph representations, have become a popular paradigm. With these models being usually characterized by intuition-driven design or highly intricate components, placing them within the theoretical analysis framework to distill the core concepts, helps understand the key principles that drive the functionality better and guide further development. Given this surge in interest, this article provides a comprehensive summary of the theoretical foundations and breakthroughs concerning the approximation and learning behaviors intrinsic to prevalent graph learning models. Encompassing discussions on fundamental aspects such as expressiveness power, generalization, optimization, and unique phenomena such as over-smoothing and over-squashing, this piece delves into the theoretical foundations and frontier driving the evolution of graph learning. In addition, this article also presents several challenges and further initiates discussions on possible solutions.

Foundations and Frontiers of Graph Learning Theory

TL;DR

This paper surveys the theoretical foundations of graph learning, focusing on three core pillars—expressive power, generalization, and optimization—and also addresses long-range interactions via over-smoothing and over-squashing. It synthesizes WL-based expressivity, higher-order and subgraph approaches, invariance/equivariance, and connections to combinatorial problems, while outlining generalization bounds (VC-dim, Rademacher, PAC-Bayes, stability, GNTK) and optimization dynamics (NTK regime, initialization, normalization, sampling). The work also discusses practical strategies to mitigate deep-GNN pathologies (skip connections, ODE-based models, graph rewiring) and outlines open questions linking theory to real-world graph tasks. Overall, it provides a structured theory of graph learning with clear directions toward more powerful, generalizable, and scalable graph models, including graph transformers and geometry-aware architectures.

Abstract

Recent advancements in graph learning have revolutionized the way to understand and analyze data with complex structures. Notably, Graph Neural Networks (GNNs), i.e. neural network architectures designed for learning graph representations, have become a popular paradigm. With these models being usually characterized by intuition-driven design or highly intricate components, placing them within the theoretical analysis framework to distill the core concepts, helps understand the key principles that drive the functionality better and guide further development. Given this surge in interest, this article provides a comprehensive summary of the theoretical foundations and breakthroughs concerning the approximation and learning behaviors intrinsic to prevalent graph learning models. Encompassing discussions on fundamental aspects such as expressiveness power, generalization, optimization, and unique phenomena such as over-smoothing and over-squashing, this piece delves into the theoretical foundations and frontier driving the evolution of graph learning. In addition, this article also presents several challenges and further initiates discussions on possible solutions.
Paper Structure (37 sections, 23 theorems, 51 equations, 1 figure, 2 tables, 2 algorithms)

This paper contains 37 sections, 23 theorems, 51 equations, 1 figure, 2 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Graph Embedding and Graph Kernels
  4. Graph Neural Networks
  5. Graph Transformer
  6. Expressive power
  7. Notations
  8. Graph isomorphism problem and WL algorithm
  9. Connect GNN with 1-WL
  10. GNNs beyond 1-WL
  11. High-order GNNs
  12. Graph property based GNNs
  13. Subgraph GNNs
  14. Non-equivariant GNNs
  15. Connect GNN with combinatorial problems
  16. ...and 22 more sections

Key Result

Theorem 1

Let $\mathcal{G}_1$ and $\mathcal{G}_2$ be any two non-isomorphic graphs. If a message passing GNN maps $\mathcal{G}_1$ and $\mathcal{G}_2$ to different embeddings, the 1-WL also decides $\mathcal{G}_1$ and $\mathcal{G}_2$ to be non-isomorphic.

Figures (1)

  • Figure 1: Illustration of 1-WL in distinguishing non-isomorphic graphs within 2 iterations.

Theorems & Definitions (32)

  • Theorem 1: Expressive power of MPNN
  • Theorem 2: Expressive power of $k$-GNN
  • Definition 1: Invariant function
  • Definition 2: Equivariant function
  • Theorem 3: Dimension of linear invariant and equivariant layers
  • Theorem 4: Expressive power $k$-order GNN
  • Theorem 5: Expressive power of PPGN
  • Definition 3: Containment-count and matching-count
  • Theorem 6: Expressive power of $k$-hop MPNN
  • Theorem 7: Expressive power of different distance metrics for solving biconnectivity problem
  • ...and 22 more