Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

mHC-GNN: Manifold-Constrained Hyper-Connections for Graph Neural Networks

Subhankar Mishra

TL;DR

mHC-GNN introduces manifold-constrained hyper-connections to graph neural networks by expanding node representations into multiple streams and constraining stream mixing to the Birkhoff polytope via Sinkhorn normalization. This architectural shift yields exponentially slower over-smoothing with a rate of $(1-\gamma)^{L/n}$ and enhances expressiveness beyond the 1-WL limit, enabling deep networks up to 128 layers while maintaining substantial accuracy. The approach is architecture-agnostic, improving a range of base GNNs across diverse datasets, with ablations confirming the necessity of the manifold constraint and demonstrating robust training stability. Practically, mHC-GNN offers a scalable, general solution to deep graph learning, producing meaningful gains on large graphs like ogbn-arxiv and enabling deeper models that better capture long-range dependencies.

Abstract

Graph Neural Networks (GNNs) suffer from over-smoothing in deep architectures and expressiveness bounded by the 1-Weisfeiler-Leman (1-WL) test. We adapt Manifold-Constrained Hyper-Connections (\mhc)~\citep{xie2025mhc}, recently proposed for Transformers, to graph neural networks. Our method, mHC-GNN, expands node representations across $n$ parallel streams and constrains stream-mixing matrices to the Birkhoff polytope via Sinkhorn-Knopp normalization. We prove that mHC-GNN exhibits exponentially slower over-smoothing (rate $(1-γ)^{L/n}$ vs.\ $(1-γ)^L$) and can distinguish graphs beyond 1-WL. Experiments on 10 datasets with 4 GNN architectures show consistent improvements. Depth experiments from 2 to 128 layers reveal that standard GNNs collapse to near-random performance beyond 16 layers, while mHC-GNN maintains over 74\% accuracy even at 128 layers, with improvements exceeding 50 percentage points at extreme depths. Ablations confirm that the manifold constraint is essential: removing it causes up to 82\% performance degradation. Code is available at \href{https://github.com/smlab-niser/mhc-gnn}{https://github.com/smlab-niser/mhc-gnn}

mHC-GNN: Manifold-Constrained Hyper-Connections for Graph Neural Networks

TL;DR

mHC-GNN introduces manifold-constrained hyper-connections to graph neural networks by expanding node representations into multiple streams and constraining stream mixing to the Birkhoff polytope via Sinkhorn normalization. This architectural shift yields exponentially slower over-smoothing with a rate of and enhances expressiveness beyond the 1-WL limit, enabling deep networks up to 128 layers while maintaining substantial accuracy. The approach is architecture-agnostic, improving a range of base GNNs across diverse datasets, with ablations confirming the necessity of the manifold constraint and demonstrating robust training stability. Practically, mHC-GNN offers a scalable, general solution to deep graph learning, producing meaningful gains on large graphs like ogbn-arxiv and enabling deeper models that better capture long-range dependencies.

Abstract

Graph Neural Networks (GNNs) suffer from over-smoothing in deep architectures and expressiveness bounded by the 1-Weisfeiler-Leman (1-WL) test. We adapt Manifold-Constrained Hyper-Connections (\mhc)~\citep{xie2025mhc}, recently proposed for Transformers, to graph neural networks. Our method, mHC-GNN, expands node representations across parallel streams and constrains stream-mixing matrices to the Birkhoff polytope via Sinkhorn-Knopp normalization. We prove that mHC-GNN exhibits exponentially slower over-smoothing (rate vs.\ ) and can distinguish graphs beyond 1-WL. Experiments on 10 datasets with 4 GNN architectures show consistent improvements. Depth experiments from 2 to 128 layers reveal that standard GNNs collapse to near-random performance beyond 16 layers, while mHC-GNN maintains over 74\% accuracy even at 128 layers, with improvements exceeding 50 percentage points at extreme depths. Ablations confirm that the manifold constraint is essential: removing it causes up to 82\% performance degradation. Code is available at \href{https://github.com/smlab-niser/mhc-gnn}{https://github.com/smlab-niser/mhc-gnn}
Paper Structure (63 sections, 11 theorems, 37 equations, 1 figure, 8 tables)

This paper contains 63 sections, 11 theorems, 37 equations, 1 figure, 8 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Work
  4. Graph Neural Networks
  5. Multi-Stream Architectures
  6. Graph Transformers
  7. Preliminaries
  8. Notation
  9. Message Passing Neural Networks
  10. Doubly Stochastic Matrices and Birkhoff Polytope
  11. Sinkhorn-Knopp Algorithm
  12. Methodology
  13. Multi-Stream Graph Representations
  14. Manifold-Constrained Hyper-Connections for GNNs
  15. Multi-Layer Propagation
  16. ...and 48 more sections

Key Result

Theorem 5.2

Consider a connected graph $G$ with normalized adjacency $\bar{\mathbf{A}}$ having spectral gap $\gamma = 1 - \lambda_2(\bar{\mathbf{A}}) > 0$. Let $\mathbf{x}_i^{(l)} \in \mathbb{R}^{n \times d}$ be the representation of node $i$ at layer $l$ in an mHC-GNN with stream mixing matrices $\{\mathcal{H} where $C$ is a constant depending on initial features and graph structure. For standard GNNs ($n=1,

Figures (1)

  • Figure 1: Extended depth analysis spanning 2 to 128 layers. Baseline GCN (red) suffers catastrophic over-smoothing beyond 16 layers, dropping to near-random performance at extreme depths. mHC-GNN (blue: n=2, green: n=4) maintains strong accuracy even at 128 layers, achieving 50+ percentage point improvements on Cora. X-axis uses log scale to visualize the full depth range. Error bands show standard deviation across 5 seeds.

Theorems & Definitions (29)

  • Definition 5.1: Over-smoothing
  • Theorem 5.2: Over-smoothing Rate in mHC-GNN
  • proof : Proof Sketch
  • Remark 5.3
  • Theorem 5.4: Expressiveness of mHC-GNN
  • proof : Proof Sketch
  • Proposition 5.5: Complexity Analysis
  • proof
  • Lemma 1.1
  • proof : Proof of Lemma \ref{['lem:standard_oversmoothing']}
  • ...and 19 more