Mitigating Oversmoothing Through Reverse Process of GNNs for Heterophilic Graphs

TL;DR

The paper addresses the over-smoothing problem in GNNs, which hampers performance on heterophilic graphs where neighboring nodes differ in label. It introduces a reverse-diffusion framework that pairs forward message-passing with an inverse process, enabling deep stacking by concatenating forward and reverse representations and employing variants based on GRAND and invertible GNNs with residual connections. The key contributions include a formal framework for reverse diffusion in GNNs, an invertible design with fixed-point inversion, and comprehensive experiments showing significant gains on heterophilic datasets while preserving performance on homophilic ones. This approach demonstrates that reversing diffusion can yield distinguishable neighborhood representations and mitigate smoothing, thus enabling effective long-range interactions in graphs. The work provides open-source code and lays groundwork for more powerful invertible GNN architectures with practical scalability.

Abstract

Graph Neural Network (GNN) resembles the diffusion process, leading to the over-smoothing of learned representations when stacking many layers. Hence, the reverse process of message passing can produce the distinguishable node representations by inverting the forward message propagation. The distinguishable representations can help us to better classify neighboring nodes with different labels, such as in heterophilic graphs. In this work, we apply the design principle of the reverse process to the three variants of the GNNs. Through the experiments on heterophilic graph data, where adjacent nodes need to have different representations for successful classification, we show that the reverse process significantly improves the prediction performance in many cases. Additional analysis reveals that the reverse mechanism can mitigate the over-smoothing over hundreds of layers. Our code is available at https://github.com/ml-postech/reverse-gnn.

Figures (6)

• Figure 1: Visualized node representations over the forward and reverse processes in Minesweeper. Top: our approach with the forward and reverse processes. Bottom: a classical GCN with a forward process only. The original node features are smoothed over the forward process, whereas the features are more distinguishable over the reverse process. Visualization details are provided in \ref{['sec:mine-exp']}.
• Figure 2: Prediction performance with varying the number of forward and reverse layers. We vary the number of layers (depth) in one direction. Due to memory constraints, we restricted the reverse depth used in Minesweeper to 256 or less.
• Figure 3: Over-smoothing levels measured by GSL over the number of layers (depth). ReP (forward) denotes the measured GSL in the forward process of GCN+ReP. We compare the results with GCN of three different depths: 16, 32, 64.
• Figure 4: Visualization of node prediction on the Minesweeper dataset. We visualize the prediction from 1) forward, 2) reverse, and 3) both representations, along with 4) the ground truth labels.
• Figure 5: The mean absolute difference between two consecutive representations in the fixed-point method.