Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks
Andreas Roth, Thomas Liebig
TL;DR
Rank collapse of node representations is identified as the root cause of over-smoothing and over-correlation in graph neural networks. The authors present a theoretical analysis showing that collapse is independent of aggregation and feature transformations, and propose the sum of Kronecker products (SKP) as a property that provably prevents rank collapse. Empirical validation on nine node-classification tasks demonstrates SKP's ability to fit data with depths up to $32$ layers, addressing limitations of traditional message-passing models. The findings suggest a paradigm shift toward preventing rank collapse and highlight the need for normalization-aware metrics to measure rank collapse in graphs.
Abstract
Our study reveals new theoretical insights into over-smoothing and feature over-correlation in graph neural networks. Specifically, we demonstrate that with increased depth, node representations become dominated by a low-dimensional subspace that depends on the aggregation function but not on the feature transformations. For all aggregation functions, the rank of the node representations collapses, resulting in over-smoothing for particular aggregation functions. Our study emphasizes the importance for future research to focus on rank collapse rather than over-smoothing. Guided by our theory, we propose a sum of Kronecker products as a beneficial property that provably prevents over-smoothing, over-correlation, and rank collapse. We empirically demonstrate the shortcomings of existing models in fitting target functions of node classification tasks.