SkipNode: On Alleviating Performance Degradation for Deep Graph Convolutional Networks

TL;DR

This work tackles deep GCN performance degradation caused by the joint effects of over-smoothing and gradient vanishing. It introduces SkipNode, a plug-and-play module that randomly skips layer convolutions for a subset of nodes in each middle layer, effectively reducing depth and improving gradient flow. The authors provide theoretical results showing SkipNode can increase the distance of layer outputs from the over-smoothing subspace and enable stronger gradient propagation, and demonstrate broad empirical gains across diverse models and graphs, including large-scale datasets. SkipNode outperforms state-of-the-art baselines, generalizes across architectures and graph types, and offers practical guidance on sampling rate strategies, making deep GCNs more robust and scalable.

Abstract

Graph Convolutional Networks (GCNs) suffer from performance degradation when models go deeper. However, earlier works only attributed the performance degeneration to over-smoothing. In this paper, we conduct theoretical and experimental analysis to explore the fundamental causes of performance degradation in deep GCNs: over-smoothing and gradient vanishing have a mutually reinforcing effect that causes the performance to deteriorate more quickly in deep GCNs. On the other hand, existing anti-over-smoothing methods all perform full convolutions up to the model depth. They could not well resist the exponential convergence of over-smoothing due to model depth increasing. In this work, we propose a simple yet effective plug-and-play module, Skipnode, to overcome the performance degradation of deep GCNs. It samples graph nodes in each convolutional layer to skip the convolution operation. In this way, both over-smoothing and gradient vanishing can be effectively suppressed since (1) not all nodes'features propagate through full layers and, (2) the gradient can be directly passed back through ``skipped'' nodes. We provide both theoretical analysis and empirical evaluation to demonstrate the efficacy of Skipnode and its superiority over SOTA baselines.

Figures (7)

• Figure 1: The sketch of how vanilla GCN and GCN with PairNorm/DropEdge/DropNode/DropMessage/SkipConnection/SkipNode perform on a graph. Vanilla GCN enables each connected node to exchange information. PairNorm renormalizes the output features. DropEdge deletes the edge between nodes 1 and 2 to make the graph sparser. DropNode-S removes node 1, while DropNode-F masks the entire input feature of node 2. SkipConnection directly adds the input on the output. SkipNode replaces the output feature of node 2 with its input.
• Figure 2: Three issues on 8/16/32/64-layer GCNs with PairNorm (PN), DropEdge (DE), DropNode (DN), DropMessage (DM), SkipConnection (SC), SkipNode-U (SNU), and SkipNode-B (SNB) using the Cora dataset. SNU/SNB refer to for SkipNode using uniform/biased sampling strategy. In (a), the features get more indistinguishable as the average MAD value from all the layers becomes smaller. In (b), only SkipNode alleviates the gradient vanishing issue. (c) shows the mean value of the L2 norm of all the model weights. Note that some curves overlap.
• Figure 3: The forward propagation scheme of SkipNode. $X^{(l)}_{tmp} = \tilde{A}X^{(l-1)}W^{(l)}$. For simplicity, we omit the nonlinear function in this figure.
• Figure 4: The effectiveness of alleviating over-smoothing. We explore (a) the log distance ratio $\log(\frac{d_{\mathcal{M}}(X^{(l)}_{2})}{d_{\mathcal{M}}(X^{(0)})})$ for each layer $l$ with varying $\rho$ and $s$ ($\rho=0$ is equivalent to vanilla GCN), and (b) the log distance ratio $\log(\frac{d_{\mathcal{M}}(X_{2})}{d_{\mathcal{M}}(X_{1})})$ for one layer calculation. All the results are averaged from $100$ runs on the same graph for each fixed combination of $\rho$ and $s$.
• Figure 5: Scalability analysis. We use three large-scale graphs under different tasks, including (a) node classification task using ogbn-arxiv (upper), ogbn-papers100M (bottom), and (b) link prediction task using ogbl-ppa. SAGE$*$ is the abbreviation for GraphSAGE_res_incep.

Theorems & Definitions (19)

• Definition 1
• Theorem 1
• proof
• Remark 1
• Lemma 1
• proof : Proof of Lemma \ref{['lem1']}
• Theorem 2: Higher Upper Bound
• proof : Proof of Theorem \ref{['thm1']}
• Theorem 3: Longer Distance from $\mathcal{M}$
• proof : Proof of Theorem \ref{['thm2']}