Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Deeper with Riemannian Geometry: Overcoming Oversmoothing and Oversquashing for Graph Foundation Models

Li Sun, Zhenhao Huang, Ming Zhang, Philip S. Yu

TL;DR

The paper tackles fundamental bottlenecks in graph foundation models by reframing MPNNs as heat-flow solvers on Riemannian manifolds and showing that global spectral-gap increases can harm gradient flow and acceleration of oversmoothing. It introduces a local, boundary-condition–driven approach that leverages local submanifold geometry to mitigate both oversquashing and oversmoothing, anchored by a Robin-condition–based Graph Boundary conditioned Neural network (GBN). The authors formalize a nonhomogeneous boundary condition and demonstrate theoretical guarantees, including distance-independent Jacobians and non-vanishing Dirichlet energy, enabling substantially deeper networks (up to 256 layers) with robust long-range information propagation. Empirically, GBN achieves state-of-the-art or competitive accuracy across homophilic and heterophilic graphs and exhibits strong performance in deep and transfer-like scenarios, highlighting its practical potential for scalable graph foundation models.

Abstract

Message Passing Neural Networks (MPNNs) is the building block of graph foundation models, but fundamentally suffer from oversmoothing and oversquashing. There has recently been a surge of interest in fixing both issues. Existing efforts primarily adopt global approaches, which may be beneficial in some regions but detrimental in others, ultimately leading to the suboptimal expressiveness. In this paper, we begin by revisiting oversquashing through a global measure -- spectral gap $λ$ -- and prove that the increase of $λ$ leads to gradient vanishing with respect to the input features, thereby undermining the effectiveness of message passing. Motivated by such theoretical insights, we propose a \textbf{local} approach that adaptively adjusts message passing based on local structures. To achieve this, we connect local Riemannian geometry with MPNNs, and establish a novel nonhomogeneous boundary condition to address both oversquashing and oversmoothing. Building on the Robin condition, we design a GBN network with local bottleneck adjustment, coupled with theoretical guarantees. Extensive experiments on homophilic and heterophilic graphs show the expressiveness of GBN. Furthermore, GBN does not exhibit performance degradation even when the network depth exceeds $256$ layers.

Deeper with Riemannian Geometry: Overcoming Oversmoothing and Oversquashing for Graph Foundation Models

TL;DR

The paper tackles fundamental bottlenecks in graph foundation models by reframing MPNNs as heat-flow solvers on Riemannian manifolds and showing that global spectral-gap increases can harm gradient flow and acceleration of oversmoothing. It introduces a local, boundary-condition–driven approach that leverages local submanifold geometry to mitigate both oversquashing and oversmoothing, anchored by a Robin-condition–based Graph Boundary conditioned Neural network (GBN). The authors formalize a nonhomogeneous boundary condition and demonstrate theoretical guarantees, including distance-independent Jacobians and non-vanishing Dirichlet energy, enabling substantially deeper networks (up to 256 layers) with robust long-range information propagation. Empirically, GBN achieves state-of-the-art or competitive accuracy across homophilic and heterophilic graphs and exhibits strong performance in deep and transfer-like scenarios, highlighting its practical potential for scalable graph foundation models.

Abstract

Message Passing Neural Networks (MPNNs) is the building block of graph foundation models, but fundamentally suffer from oversmoothing and oversquashing. There has recently been a surge of interest in fixing both issues. Existing efforts primarily adopt global approaches, which may be beneficial in some regions but detrimental in others, ultimately leading to the suboptimal expressiveness. In this paper, we begin by revisiting oversquashing through a global measure -- spectral gap -- and prove that the increase of leads to gradient vanishing with respect to the input features, thereby undermining the effectiveness of message passing. Motivated by such theoretical insights, we propose a \textbf{local} approach that adaptively adjusts message passing based on local structures. To achieve this, we connect local Riemannian geometry with MPNNs, and establish a novel nonhomogeneous boundary condition to address both oversquashing and oversmoothing. Building on the Robin condition, we design a GBN network with local bottleneck adjustment, coupled with theoretical guarantees. Extensive experiments on homophilic and heterophilic graphs show the expressiveness of GBN. Furthermore, GBN does not exhibit performance degradation even when the network depth exceeds layers.
Paper Structure (60 sections, 10 theorems, 113 equations, 6 figures, 8 tables, 1 algorithm)

This paper contains 60 sections, 10 theorems, 113 equations, 6 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Related Work
  4. Oversmoothing.
  5. Oversquashing.
  6. Riemannian Geometry & MPNNs.
  7. Preliminaries
  8. Riemannian Manifold & Heat Flow.
  9. MPNN, Oversquashing & Oversmoothing.
  10. Theoretical Insights on Oversquashing
  11. A unified formalism.
  12. On oversquashing: revisiting the spectral gap
  13. On oversmoothing.
  14. A more intuitive explanation.
  15. Local Riemannian Geometry for the Best of Both Worlds
  16. ...and 45 more sections

Key Result

Theorem 4.1

If the Ricci curvature of manifold $\mathcal{M}$ is bounded by $-(n-1)\xi^2$, and $\xi \geq 0$, then $\frac{h^2_\mathcal{M}}{4} \leq \lambda_1 < C(\xi \cdot h_\mathcal{M} + h^2_\mathcal{M}),$ where $C$ is a constant.

Figures (6)

  • Figure 1: Illustration of different solutions against oversquashing. (A) In typical MPNNs, messages are "squashed" at the bottleneck of the graph structure. (B) In graph rewriting (e.g., BORF icml23-BORF), oversquashing at the bottleneck is mitigated by adding edges, but the removal of edges (in light of oversmoothing) hinders the message-passing in other regions. (C) In the proposed GBN, an adaptive adjustment based on the local Riemannian geometry is introduced to message passing in each region to enhance expressiveness.
  • Figure 2: A visual illustration.
  • Figure 3: Dirichlet energy curve (lg-lg).
  • Figure 5: Results of graph transfer task in terms of Mean Squared Error (%).
  • Figure 6: Case study
  • ...and 1 more figures

Theorems & Definitions (19)

  • Theorem 4.1: Cheeger and Buser's Inequality 1982A
  • Theorem 4.2: Spectral Gap and Gradient Vanishing, \ref{['prf:Spectral Gap and Gradient Vanishing']}
  • Theorem 4.3: Spectral Gap and Dirichlet Energy, \ref{['prf:Spectral Gap and Dirichlet Energy']}
  • Theorem 5.1: Oversquashing under Boundary Conditions, \ref{['prf:Oversquashing under Boundary Conditions']}
  • Corollary 5.2: Adaptivity of Robin Condition, \ref{['prf:Adaptivity of Robin Condition']}
  • Lemma 5.3: Oversmoothing under Source Terms, \ref{['prf:Oversmoothing under Source Terms']}
  • Theorem 5.4: Equivalence, \ref{['prf:Equivalence']}
  • Definition 6.1: Boundary Nodes
  • Lemma 6.2: Solution to Heat Equation System, \ref{['prf:Solution to Heat Equation System']}
  • Theorem 6.3: Distance Independence, \ref{['prf:Distance Independence']}
  • ...and 9 more