Unveiling Mode Connectivity in Graph Neural Networks

TL;DR

This work investigates mode connectivity in Graph Neural Networks (GNNs), revealing a distinct non-linear connectivity pattern largely governed by graph structure rather than architectural choices. By analyzing 12 diverse graphs and employing linear and quadratic Bézier interpolations between independently trained minima, the authors show that minima are often connected through low-barrier nonlinear paths, with the barrier magnitude correlating with graph properties such as density, homophily, and feature separability. A theoretical bound ties the loss barrier to the spectral gap of the graph, providing principled explanations for when connectivity is strong and how it impacts generalization. The paper further introduces a Wasserstein-distance based metric on mode-connectivity curves to quantify cross-domain graph similarity and bound transferability gaps, offering a practical diagnostic for domain alignment and graph-based knowledge transfer. Overall, the results bridge optimization geometry with graph structure, informing better GNN training and transfer strategies across domains.

Abstract

A fundamental challenge in understanding graph neural networks (GNNs) lies in characterizing their optimization dynamics and loss landscape geometry, critical for improving interpretability and robustness. While mode connectivity, a lens for analyzing geometric properties of loss landscapes has proven insightful for other deep learning architectures, its implications for GNNs remain unexplored. This work presents the first investigation of mode connectivity in GNNs. We uncover that GNNs exhibit distinct non-linear mode connectivity, diverging from patterns observed in fully-connected networks or CNNs. Crucially, we demonstrate that graph structure, rather than model architecture, dominates this behavior, with graph properties like homophily correlating with mode connectivity patterns. We further establish a link between mode connectivity and generalization, proposing a generalization bound based on loss barriers and revealing its utility as a diagnostic tool. Our findings further bridge theoretical insights with practical implications: they rationalize domain alignment strategies in graph learning and provide a foundation for refining GNN training paradigms.

Figures (11)

• Figure 1: Performance of linear interpolations between two minima on eight real-world datasets. The x-axis represents the interpolation coefficient $\alpha$, and the y-axis shows train accuracy, test accuracy, train loss, and test loss.
• Figure 2: Performance of Quadratic Bézier Curve Interpolations Between Two Minima. This figure shows train/test loss and accuracy curves for quadratic Bézier curve interpolations across eight datasets. Compared to linear interpolation (Figure \ref{['fig: figure1']}), Bézier curves better bypass loss barriers, indicating that GNN minima, while not always linearly connected, often lie on a smooth low-loss manifold.
• Figure 3: Contour visualization of loss basin
• Figure 4: Comparison between the loss barrier of GNN and MLP.
• Figure 5: The trend of mode connectivity, as measured by the barrier, with changes in graph properties.

Theorems & Definitions (8)

• Definition 2.1: Contextual Stochastic Block Model (CSBM)
• Definition 2.2: Loss Barrier
• Definition 3.1: Spectral Gap chung1997spectral
• Theorem 3.2: General Bound of the Loss Barrier in $L$-Layer GNNs
• Remark 3.3: Implications for Mode Connectivity
• Corollary 3.4: Graph Property Perspective of the Loss Barrier
• Proposition 3.5: Loss Barrier in Node Classification on CSBM Datasets
• Theorem 4.1: Generalization Bound via Loss Barrier