Covered Forest: Fine-grained generalization analysis of graph neural networks

TL;DR

This work tackles the problem of generalization for message-passing graph neural networks by introducing refined graph similarities that align with MPNN computations. By leveraging data-dependent coverings, Tree distance, Forest distance, and the Tree Mover's distance (TMD), the authors develop a robustness framework and prove generalization bounds that hold under a range of losses, not just the 0-1 loss, and for various aggregation schemes including sum and mean. A key contribution is showing that graph structure and the choice of metric (e.g., Tree distance vs WL-based metrics) significantly affect covering numbers and hence generalization bounds, with empirical results validating the theory on synthetic graph families and real-world datasets. The paper also introduces the mean-1-WL (1-MWL) perspective and connects it to mean-aggregation MPNNs, offering fine-grained analyses that unify expressivity, robustness, and generalization, and yielding practical guidance on architecture and metric selection for improved generalization in graph learning.

Abstract

The expressive power of message-passing graph neural networks (MPNNs) is reasonably well understood, primarily through combinatorial techniques from graph isomorphism testing. However, MPNNs' generalization abilities -- making meaningful predictions beyond the training set -- remain less explored. Current generalization analyses often overlook graph structure, limit the focus to specific aggregation functions, and assume the impractical, hard-to-optimize $0$-$1$ loss function. Here, we extend recent advances in graph similarity theory to assess the influence of graph structure, aggregation, and loss functions on MPNNs' generalization abilities. Our empirical study supports our theoretical insights, improving our understanding of MPNNs' generalization properties.

Figures (11)

• Figure 1: Illustrating how the choice of pseudo-metrics influences the geometry and the size of coverings, leading to a tighter generalization analysis.
• Figure 2: An illustration of the unrolling trees and the process of obtaining the mean unrolling trees from the unrolling trees through mean pruning.
• Figure 3: An illustration of the computation of the Forest distance for depth $L=2$ for two labeled graphs $G_1$ and $G_2$. Grey vertices indicate the padded vertices in the unrollings, and $\varphi$ represents an edge-preserving bijection between the two forests.
• Figure 4: The covering number $\mathcal{N}(\cdot, \mathrm{FD}_{3}, \varepsilon)$ and the number of $1$-WL indistinguishable graphs $m_n$ (upper row, black) for the graph families $\mathcal{G}_n$, $\mathcal{T}_n^{(2)}$, and $\mathcal{F}_n$ across different graph sizes $n$ and radii $\varepsilon$.
• Figure 5: The covering number $\mathcal{N}(\cdot, \mathrm{FD}_{3}, \varepsilon)$ and the number of $1$-WL indistinguishable graphs $m_n$ (upper row, black) for real-world datasets across varying graph sizes $n$ and radii $\varepsilon$.

Theorems & Definitions (80)

• Lemma 1: Folklore, see, e.g., Mor+2020
• Proposition 2
• Proposition 3
• Lemma 4
• Lemma 5
• Lemma 6
• Theorem 7: Theorem 3 in Xu+2012
• Theorem 8: Theorem 3 in Kaw+2022
• Definition 9
• Theorem 10