Graph Representational Learning: When Does More Expressivity Hurt Generalization?

TL;DR

This paper tackles the question of when increasing GNN expressivity improves or hurts generalization. It introduces task-aligned pseudometrics, the $\zeta$-Tree Mover Distances (\zeta-TMDs), to quantify structure–label alignment and proves Lipschitz continuity of $\zeta$-MPNNs with respect to these distances. It then derives data-dependent PAC-Bayes generalization bounds that decompose into a model-complexity term and a structural similarity term, where the latter depends on the distance between training and test graphs under $\zeta$-TMD$^{T+1}$. The results show that larger expressivity helps only when it enhances the alignment between the training and test structures; otherwise, it can degrade generalization due to increased structural discrepancy. Empirical experiments on synthetic and real graph datasets corroborate the theory, showing that task-aligned, moderately expressive GNNs can outperform highly expressive ones and that generalization deteriorates as test graphs become structurally distant from the training set.

Abstract

Graph Neural Networks (GNNs) are powerful tools for learning on structured data, yet the relationship between their expressivity and predictive performance remains unclear. We introduce a family of premetrics that capture different degrees of structural similarity between graphs and relate these similarities to generalization, and consequently, the performance of expressive GNNs. By considering a setting where graph labels are correlated with structural features, we derive generalization bounds that depend on the distance between training and test graphs, model complexity, and training set size. These bounds reveal that more expressive GNNs may generalize worse unless their increased complexity is balanced by a sufficiently large training set or reduced distance between training and test graphs. Our findings relate expressivity and generalization, offering theoretical insights supported by empirical results.

Figures (11)

• Figure 1: Left: Train–test errors for several GNN variants on a synthetic cycle-counting task; moderately expressive models such as $\mathcal{F}_4$‑MPNN barcelo_gnns_local_graph_params, i.e., MPNNs augmented with cycle counts, generalize best, while more expressive ones tend to overfit. Center: Training-loss curves of a MPNN on Mutagenicity under increasing label noise $p$. Right: Corresponding test errors on BZR, Mutagenicity, and NCI109 rises sharply as label-structure correlation is essential for generalization (mean $\pm$ standard deviation across five seeds). See \ref{['sec:label_noise']} for more details.
• Figure 2: Accuracy of a GIN with 1, 3, and 5 layers versus TMD (log scale) to the training dataset.
• Figure 3: Error‑bound curves for Mutagenicity, and NCI109. Each plot shows our theoretical bound (blue, left axis) and the empirical generalization error (red, right axis) as a function of TMD to the training set. Shaded areas indicate $\pm$1 standard deviation across 10 random splits.
• Figure 4: Left: Training-loss trajectories of a GIN on Mutagenicity under increasing label noise $p$. Center: Corresponding test errors on Mutagenicityrises sharply as label–structure correlation is essential for generalization (mean $\pm$ standard deviation across five seeds). Right: Number of epochs to overfit, i.e., reach 99% training accuracy under increasing label noise $p$.
• Figure 5: Left: Training-loss trajectories of a GIN on BZR under increasing label noise $p$. Center: Corresponding test errors on BZRrises sharply as label–structure correlation is essential for generalization (mean $\pm$ standard deviation across five seeds). Right: Number of epochs to overfit, i.e., reach 99% training accuracy under increasing label noise $p$.

Theorems & Definitions (44)

• Definition 2.1: Graph Invariant and CRA
• Definition 2.2: Message Passing Neural Networks (MPNNs)
• Definition 3.1
• Proposition 3.2
• Proposition 3.3
• Theorem 3.4
• Corollary 3.5
• Theorem 4.1
• Theorem 5.1
• Definition B.1