Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Higher-Order Graphon Neural Networks: Approximation and Cut Distance

Daniel Herbst, Stefanie Jegelka

TL;DR

This work extends higher-order GNNs to graphons by formulating graphon-signals and signal-weighted homomorphism densities, enabling a principled analysis of expressivity, continuity, and transferability. It introduces Invariant Graphon Networks (IWNs), built with bounded linear equivariant layers, and proves they are $k$-WL expressive and universal for $L^p$-distance approximation on compact graphon-signal spaces. The authors also show that typical higher-order GNNs are discontinuous in the cut distance due to $k$-WL edge-weight distributions, yet transferability persists for a broad class of IWNs through WL-measure representations and graphon-signal sampling theory. The results connect graphon limit theory with higher-order GNNs, align IWNs with IGN-small, and include a transferability analysis with quantitative rates, complemented by toy experiments illustrating convergence behavior and stability across graph sizes.

Abstract

Graph limit models, like graphons for limits of dense graphs, have recently been used to study size transferability of graph neural networks (GNNs). While most literature focuses on message passing GNNs (MPNNs), in this work we attend to the more powerful higher-order GNNs. First, we extend the $k$-WL test for graphons (Böker, 2023) to the graphon-signal space and introduce signal-weighted homomorphism densities as a key tool. As an exemplary focus, we generalize Invariant Graph Networks (IGNs) to graphons, proposing Invariant Graphon Networks (IWNs) defined via a subset of the IGN basis corresponding to bounded linear operators. Even with this restricted basis, we show that IWNs of order $k$ are at least as powerful as the $k$-WL test, and we establish universal approximation results for graphon-signals in $L^p$ distances. This significantly extends the prior work of Cai & Wang (2022), showing that IWNs--a subset of their IGN-small--retain effectively the same expressivity as the full IGN basis in the limit. In contrast to their approach, our blueprint of IWNs also aligns better with the geometry of graphon space, for example facilitating comparability to MPNNs. We highlight that, while typical higher-order GNNs are discontinuous w.r.t. cut distance--which causes their lack of convergence and is inherently tied to the definition of $k$-WL--transferability remains achievable.

Higher-Order Graphon Neural Networks: Approximation and Cut Distance

TL;DR

This work extends higher-order GNNs to graphons by formulating graphon-signals and signal-weighted homomorphism densities, enabling a principled analysis of expressivity, continuity, and transferability. It introduces Invariant Graphon Networks (IWNs), built with bounded linear equivariant layers, and proves they are -WL expressive and universal for -distance approximation on compact graphon-signal spaces. The authors also show that typical higher-order GNNs are discontinuous in the cut distance due to -WL edge-weight distributions, yet transferability persists for a broad class of IWNs through WL-measure representations and graphon-signal sampling theory. The results connect graphon limit theory with higher-order GNNs, align IWNs with IGN-small, and include a transferability analysis with quantitative rates, complemented by toy experiments illustrating convergence behavior and stability across graph sizes.

Abstract

Graph limit models, like graphons for limits of dense graphs, have recently been used to study size transferability of graph neural networks (GNNs). While most literature focuses on message passing GNNs (MPNNs), in this work we attend to the more powerful higher-order GNNs. First, we extend the -WL test for graphons (Böker, 2023) to the graphon-signal space and introduce signal-weighted homomorphism densities as a key tool. As an exemplary focus, we generalize Invariant Graph Networks (IGNs) to graphons, proposing Invariant Graphon Networks (IWNs) defined via a subset of the IGN basis corresponding to bounded linear operators. Even with this restricted basis, we show that IWNs of order are at least as powerful as the -WL test, and we establish universal approximation results for graphon-signals in distances. This significantly extends the prior work of Cai & Wang (2022), showing that IWNs--a subset of their IGN-small--retain effectively the same expressivity as the full IGN basis in the limit. In contrast to their approach, our blueprint of IWNs also aligns better with the geometry of graphon space, for example facilitating comparability to MPNNs. We highlight that, while typical higher-order GNNs are discontinuous w.r.t. cut distance--which causes their lack of convergence and is inherently tied to the definition of -WL--transferability remains achievable.

Paper Structure

This paper contains 68 sections, 46 theorems, 150 equations, 6 figures, 1 table, 3 algorithms.

Table of Contents

  1. Introduction
  2. Background
  3. General Background on Graphon Theory
  4. Graphons.
  5. Discretization and sampling.
  6. Homomorphism densities.
  7. $k$-WL and Homomorphism Expressivity
  8. Extension to Graphon-Signals
  9. Signal-Weighted Homomorphism Densities
  10. Invariant Graphon Networks
  11. Linear Equivariant Layers
  12. A canonical basis of $\textnormal{LE}_{{k} \to {\ell}}$.
  13. Definition of Invariant Graphon Networks
  14. Expressivity of Invariant Graphon Networks
  15. Cut Distance and Transferability of Higher-Order WNNs
  16. ...and 53 more sections

Key Result

Theorem 3.2

Fix $r>1$ and let $(W, f), (V, g) \in \mathcal{WS}_{r}$. Then, the following statements are equivalent:

Figures (6)

  • Figure 1: Example graphons $W$.
  • Figure 2: Absolute output errors $\mathcal{N}^{}_{}(W)$ vs. $\mathcal{N}^{}_{}(\mathbb{G}_n(W))$.
  • Figure 3: $90\%$ pred. interval widths of $\mathcal{N}^{}_{}(\mathbb{G}_n(W))$.
  • Figure 4: Equivalence chain for the proof of \ref{['thm:cut_distance_equivalences']}.
  • Figure 5: A nice tree decomposition for a node-featured graph of treewidth 2, along with corresponding atomic tri-labeled graphs.
  • ...and 1 more figures

Theorems & Definitions (85)

  • Definition 3.1: Signal-weighted homomorphism density
  • Theorem 3.2: Characterizations of weak isomorphism for graphon-signals
  • Corollary 3.2: Convergence in graphon-signal space
  • Theorem 3.3: $k$-WL for graphon-signals, informal
  • Definition 4.1: Linear equivariant layer
  • Theorem 4.2
  • Definition 4.3: In- and equivariant graphon networks
  • Proposition 4.3
  • Lemma 4.3
  • Theorem 4.4: Approximation of signal-weighted homomorphism densities
  • ...and 75 more