Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-convolutional Graph Neural Networks

Yuanqing Wang, Kyunghyun Cho

TL;DR

This work addresses fundamental limitations of convolutional GNNs, including limited expressiveness, over-smoothing, and over-squashing. It proposes a non-convolutional framework, named random walk with unifying memory (RUM), that leverages stochastic random walks terminating at each node and unifies semantic and topological walk information through a GRU-based aggregation. Theoretical results show RUM is permutation equivariant and strictly more expressive than the WL test, capable of distinguishing non-isomorphic graphs as walk length grows, while mitigating smoothing and squashing via non-contractive mappings. Empirically, RUM attains competitive or superior performance across node- and graph-level tasks, exhibits robustness and scalability, and often surpasses simple convolutional GNNs in speed on GPUs, albeit with limitations on very dense graphs. Overall, RUM provides a scalable, expressive, and efficient alternative to convolutional GNNs with strong potential for long-range graph reasoning and applications in chemistry and complex networks.

Abstract

Rethink convolution-based graph neural networks (GNN) -- they characteristically suffer from limited expressiveness, over-smoothing, and over-squashing, and require specialized sparse kernels for efficient computation. Here, we design a simple graph learning module entirely free of convolution operators, coined random walk with unifying memory (RUM) neural network, where an RNN merges the topological and semantic graph features along the random walks terminating at each node. Relating the rich literature on RNN behavior and graph topology, we theoretically show and experimentally verify that RUM attenuates the aforementioned symptoms and is more expressive than the Weisfeiler-Lehman (WL) isomorphism test. On a variety of node- and graph-level classification and regression tasks, RUM not only achieves competitive performance, but is also robust, memory-efficient, scalable, and faster than the simplest convolutional GNNs.

Non-convolutional Graph Neural Networks

TL;DR

This work addresses fundamental limitations of convolutional GNNs, including limited expressiveness, over-smoothing, and over-squashing. It proposes a non-convolutional framework, named random walk with unifying memory (RUM), that leverages stochastic random walks terminating at each node and unifies semantic and topological walk information through a GRU-based aggregation. Theoretical results show RUM is permutation equivariant and strictly more expressive than the WL test, capable of distinguishing non-isomorphic graphs as walk length grows, while mitigating smoothing and squashing via non-contractive mappings. Empirically, RUM attains competitive or superior performance across node- and graph-level tasks, exhibits robustness and scalability, and often surpasses simple convolutional GNNs in speed on GPUs, albeit with limitations on very dense graphs. Overall, RUM provides a scalable, expressive, and efficient alternative to convolutional GNNs with strong potential for long-range graph reasoning and applications in chemistry and complex networks.

Abstract

Rethink convolution-based graph neural networks (GNN) -- they characteristically suffer from limited expressiveness, over-smoothing, and over-squashing, and require specialized sparse kernels for efficient computation. Here, we design a simple graph learning module entirely free of convolution operators, coined random walk with unifying memory (RUM) neural network, where an RNN merges the topological and semantic graph features along the random walks terminating at each node. Relating the rich literature on RNN behavior and graph topology, we theoretically show and experimentally verify that RUM attenuates the aforementioned symptoms and is more expressive than the Weisfeiler-Lehman (WL) isomorphism test. On a variety of node- and graph-level classification and regression tasks, RUM not only achieves competitive performance, but is also robust, memory-efficient, scalable, and faster than the simplest convolutional GNNs.
Paper Structure (49 sections, 4 theorems, 23 equations, 6 figures, 9 tables, 1 algorithm)

This paper contains 49 sections, 4 theorems, 23 equations, 6 figures, 9 tables, 1 algorithm.

Table of Contents

  1. Introduction: Convolutions in GNNs
  2. Limited expressiveness. (Figure \ref{['fig:cycle']})
  3. Over-smoothing. (Figure \ref{['fig:dirichlet']})
  4. Over-squashing. (Figure \ref{['fig:bottle']})
  5. Main contributions: Non-convolutional GNNs as a joint remedy.
  6. Related works: ways to walk on a graph
  7. Walk-based GNNs.
  8. Random walk kernel GNNs.
  9. State-of-the-art methods to alleviate oversmoothing.
  10. Architecture: combining topologic and semantic trajectories of walks
  11. Random walks on graphs.
  12. Anonymous experiment.
  13. Unifying memory: combining semantic and topological representations.
  14. Layer choices.
  15. Runtime complexity.
  16. ...and 34 more sections

Key Result

Theorem 4

Up to the Reconstruction Conjecture kelly1957congruence, RUM with sufficiently long $l$-step random walks can distinguish non-isomorphic graphs satisfying Assumption assumption:graph.

Figures (6)

  • Figure 1: RUM can (in closed form), whereas the Weisfeiler-Lehman (WL) isomorphism test and WL-equivalent GNNs cannot, distinguish these two graphs---an illustration of Example \ref{['eg:cycle']}.
  • Figure 2: RUM alleviates over-smoothing. Dirichlet energy ($\mathcal{E}$) on Cora DBLP:journals/corr/YangCS16 graph plotted against $L$, the number of steps or layers.
  • Figure 3: Impact of number of samples and walk length. Test classification accuracy of Cora DBLP:journals/corr/YangCS16 with varying numbers of samples and walk length.
  • Figure 4: RUM is faster than convolutional GNNs on GPU. Inference time over the Cora DBLP:journals/corr/YangCS16 graph on CPU and CUDA devices, respectively, plotted against $L$, the number of message-passing steps or equivalently the length of random walks. Numbers in the bracket indicate the number of sampled random walks drawn.
  • Figure 5: Long-range neighborhood matching training accuracy $\uparrow$DBLP:journals/corr/abs-2006-05205 with 32 unit models.
  • ...and 1 more figures

Theorems & Definitions (11)

  • Remark 1: Permutation equivariance
  • Theorem 4: RUM can distinguish non-isomorphic graphs
  • Corollary 4.1: RUM is more expressive than WL-test
  • Definition 5
  • Lemma 6: RUM alleviates over-smoothing.
  • Lemma 7: RUM attenuates over-squashing
  • Remark 8: Inequality in distribution
  • Example 8.1: Cycle detection.
  • Example 8.2: Diameter.
  • proof
  • ...and 1 more