Resolving Oversmoothing with Opinion Dissensus

TL;DR

The paper addresses oversmoothing in deep graph neural networks by reframing GNNs as opinion dynamics and showing that linear models inherently converge to consensus. It introduces a nonlinear, behavior-inspired continuous-depth GNN called BIMP that uses nonlinear opinion dynamics to induce dissensus, thereby resisting oversmoothing and enabling stable learning at great depths. The main contributions include a formal GNN–opinion dynamics analogy, a provably dissensus-capable nonlinear OD model, and extensive empirical validation across homophilic, heterophilic, and large-scale datasets, with well-behaved gradients. The work offers a principled approach to designing deep GNNs that preserve discriminative power in complex graph structures, potentially impacting applications requiring robust deep graph representations.

Abstract

While graph neural networks (GNNs) have allowed researchers to successfully apply neural networks to non-Euclidean domains, deep GNNs often exhibit lower predictive performance than their shallow counterparts. This phenomena has been attributed in part to oversmoothing, the tendency of node representations to become increasingly similar with network depth. In this paper we introduce an analogy between oversmoothing in GNNs and consensus (i.e., perfect agreement) in the opinion dynamics literature. We show that the message passing algorithms of several GNN models are equivalent to linear opinion dynamics models which have been shown to converge to consensus for all inputs regardless of the graph structure. This new perspective on oversmoothing motivates the use of nonlinear opinion dynamics as an inductive bias in GNN models. In our Behavior-Inspired Message Passing (BIMP) GNN, we leverage the nonlinear opinion dynamics model which is more general than the linear opinion dynamics model, and can be designed to converge to dissensus for general inputs. Through extensive experiments we show that BIMP resists oversmoothing beyond 100 time steps and consistently outperforms existing architectures even when those architectures are amended with oversmoothing mitigation techniques. We also show that BIMP has several desirable properties including well behaved gradients and adaptability to homophilic and heterophilic datasets.

Figures (4)

• Figure 1: Nonlinear opinion dynamics and dissensus.(Left) The pitchfork bifurcation diagram illustrates a change in the number and stability of opinion states with the attention parameter $u$ (stable equilibria are illustrated with a solid line and unstable equilibria are illustrated with a dotted line). In the diagram, $z$ represents the weighted average of agent opinions, and $u^*$ represents the bifurcation point. When the input term $b=0$ we have the pitchfork bifurcation (top-left), and when $b>0$ we have its unfolding (bottom-left). (Right) The time evolution of agent opinions under the nonlinear opinion dynamics model depends on the initial weighted average of agent opinions $z$, the attention parameter $u$, and the weighted average of agent inputs $b$. Each subfigure corresponds to an initial condition on the left. In all cases, the initial $z$ is the same. (a) ($u<u^*$, $b=0$). Agent opinions converge to a neutral consensus (i.e., perfect agreement) which is equivalent to oversmoothing. (b) ($u>u^{*}$, $b=0$). Agent opinions converge to dissensus with low variance, $z$ is positive. (c) ($u=u^{*}$, $b=0$). Agent opinions converge to a neutral consensus. (d) ($u=u^{*}$, $b>0$). Agent opinions converge to dissensus with high variance, $z$ is positive.
• Figure 2: Classification accuracy and Dirichlet energy. BIMP is designed to learn node representations that resist oversmoothing even for very large depths. (Left) We compare the classification accuracy of BIMP to baseline models for architectures with $1, 2, 4, 8, 16, 32, 64$ and $128$ timesteps. Our BIMP model is stable out to 128 timesteps, while baseline performance deteriorates after 32 timesteps. (Right) We compare the Dirichlet energy of node features over a range of network depths. The Dirichlet energy of BIMP remains stable even at very deep layers, while the energy of baseline modes does not.
• Figure 3: Classification accuracy. BIMP is designed to learn node representations that resist oversmoothing even for very large depths. We compare the classification accuracy of BIMP to baseline models for architectures with $1, 2, 4, 8, 16, 32, 64$ and $128$ timesteps. Our BIMP model and its variants are stable out to 128 timesteps, while baseline performance deteriorates after 32 timesteps.
• Figure 4: Dirichlet energy. BIMP is designed to learn node representations that resist oversmoothing even for very large depths. We compare the Dirichlet energy of node features over a range of network depths. The Dirichlet energy of BIMP remains stable even at very deep layers, while the energy of baseline modes does not.

Theorems & Definitions (35)

• Lemma 4.1: Linear dynamics oversmooth
• Lemma 4.2: Laplacian dynamics oversmooth
• Lemma 4.3: Laplacian dynamics with an external input oversmooth
• Definition 5.1: Effective adjacency matrix
• Lemma 5.2
• Lemma 5.3
• Lemma 5.4: Bifurcation point $u^*$
• Lemma 5.5: BIMP converges to equilibrium
• Theorem 5.6: Dissensus in BIMP
• Theorem 5.7: BIMP has well behaved gradients