Measuring Over-smoothing beyond Dirichlet energy
Weiqi Guan, Zihao Shi
TL;DR
This paper broadens the study of over-smoothing in graph neural networks by introducing a generalized family of higher-order node-similarity measures E_m that capture derivatives beyond the first order. It establishes equivalence across these measures for m>=1, relates them to the Poincaré inequality, and provides a spectral decomposition showing how higher-order energies emphasize high-frequency components. Theoretical results yield sharp decay rates for both continuous heat diffusion and discrete random-walk propagation, linking over-smoothing to the graph Laplacian's spectral gap and highlighting a trade-off with over-squashing. Empirical tests on standard datasets demonstrate significant over-smoothing of attention-based GNNs when evaluated with these refined metrics, suggesting the need for diffusion-aware design in deep GNNs.
Abstract
While Dirichlet energy serves as a prevalent metric for quantifying over-smoothing, it is inherently restricted to capturing first-order feature derivatives. To address this limitation, we propose a generalized family of node similarity measures based on the energy of higher-order feature derivatives. Through a rigorous theoretical analysis of the relationships among these measures, we establish the decay rates of Dirichlet energy under both continuous heat diffusion and discrete aggregation operators. Furthermore, our analysis reveals an intrinsic connection between the over-smoothing decay rate and the spectral gap of the graph Laplacian. Finally, empirical results demonstrate that attention-based Graph Neural Networks (GNNs) suffer from over-smoothing when evaluated under these proposed metrics.