RandAlign: A Parameter-Free Method for Regularizing Graph Convolutional Networks

TL;DR

This work tackles the over-smoothing problem in graph convolutional networks by introducing RandAlign, a parameter-free stochastic regularization that randomly aligns a layer's embeddings with the previous layer through norm-matched random interpolation. By integrating a scaling step prior to interpolation and using a residual-style update, RandAlign preserves the advantages of graph convolution while reducing embeddings' smoothness, improving generalization and optimization stability. Empirical results on seven diverse graph benchmarks show that RandAlign consistently enhances performance across multiple base GCN architectures and tasks, sometimes achieving state-of-the-art outcomes. The method's plug-and-play nature and lack of extra trainable parameters make it a practical and broadly applicable tool for robust graph representation learning.

Abstract

Studies continually find that message-passing graph convolutional networks suffer from the over-smoothing issue. Basically, the issue of over-smoothing refers to the phenomenon that the learned embeddings for all nodes can become very similar to one another and therefore are uninformative after repeatedly applying message passing iterations. Intuitively, we can expect the generated embeddings become smooth asymptotically layerwisely, that is each layer of graph convolution generates a smoothed version of embeddings as compared to that generated by the previous layer. Based on this intuition, we propose RandAlign, a stochastic regularization method for graph convolutional networks. The idea of RandAlign is to randomly align the learned embedding for each node with that of the previous layer using randomly interpolation in each graph convolution layer. Through alignment, the smoothness of the generated embeddings is explicitly reduced. To better maintain the benefit yielded by the graph convolution, in the alignment step we introduce to first scale the embedding of the previous layer to the same norm as the generated embedding and then perform random interpolation for aligning the generated embedding. RandAlign is a parameter-free method and can be directly applied without introducing additional trainable weights or hyper-parameters. We experimentally evaluate RandAlign on different graph domain tasks on seven benchmark datasets. The experimental results show that RandAlign is a general method that improves the generalization performance of various graph convolutional network models and also improves the numerical stability of optimization, advancing the state of the art performance for graph representation learning.

Figures (2)

• Figure 1: An illustrative example for understanding the over-smoothing issue. We consider a two node fully connected graph and use a GAT model that layerwisely learn embeddings using the equation $\mathbf{h}_{i}^{(k)} = \sum_{v \in \mathcal{N}(u)} \alpha_{u,v} \mathbf{h}_{v}^{(k-1)}$, wherein $\alpha_{u,v}>0$ and $\sum_{v \in \mathcal{N}(u)} \alpha_{u,v}=1$. We have simplified the model by removing the non-linearity and learnable parameter matrix. We show that the learned embeddings layerwisely become smoothed than the previous layer due to the convex combination of neighbourhood features.
• Figure 2: Learning curves on CIFAR10. Bold lines are training curves and thin lines are test curves. We show that our RandAlign method improves the generalization performance by reducing the issue of over-smoothing.