Rethinking Oversmoothing in Graph Neural Networks: A Rank-Based Perspective
Kaicheng Zhang, Piero Deidda, Desmond Higham, Francesco Tudisco
TL;DR
This work tackles oversmoothing in graph neural networks by challenging the primacy of energy-based metrics such as Dirichlet energy, which can be misleading in realistic nonlinear settings. It proposes rank-based measures, notably numerical rank and effective rank, as robust indicators of oversmoothing, and proves theoretically that the numerical rank converges to 1 for broad classes of GNNs under nonnegative weights, extending beyond linear models to nonlinear activations via nonlinear Perron–Frobenius theory. The authors provide extensive experiments across diverse GNN architectures and datasets, demonstrating that rank relaxations track performance degradation more reliably than energy-based metrics, which may remain flat even as accuracy drops. This rank-centric perspective offers a scale-invariant, eigenspace-agnostic tool for diagnosing and mitigating oversmoothing in practical deep GNNs, with potential implications for architecture design and regularization strategies.
Abstract
Oversmoothing is a fundamental challenge in graph neural networks (GNNs): as the number of layers increases, node embeddings become increasingly similar, and model performance drops sharply. Traditionally, oversmoothing has been quantified using metrics that measure the similarity of neighbouring node features, such as the Dirichlet energy. While these metrics are related to oversmoothing, we argue they have critical limitations and fail to reliably capture oversmoothing in realistic scenarios. For instance, they provide meaningful insights only for very deep networks and under somewhat strict conditions on the norm of network weights and feature representations. As an alternative, we propose measuring oversmoothing by examining the numerical or effective rank of the feature representations. We provide theoretical support for this approach, demonstrating that the numerical rank of feature representations converges to one for a broad family of nonlinear activation functions under the assumption of nonnegative trained weights. To the best of our knowledge, this is the first result that proves the occurrence of oversmoothing in the nonlinear setting without assumptions on the boundedness of the weight matrices. Along with the theoretical findings, we provide extensive numerical evaluation across diverse graph architectures. Our results show that rank-based metrics consistently capture oversmoothing, whereas energy-based metrics often fail. Notably, we reveal that a significant drop in the rank aligns closely with performance degradation, even in scenarios where energy metrics remain unchanged.