When Do Graph Neural Networks Help with Node Classification? Investigating the Impact of Homophily Principle on Node Distinguishability

TL;DR

The paper investigates when graph neural networks (GNNs) help node classification beyond the homophily assumption by introducing Contextual Stochastic Block Model for Homophily (CSBM-H), which explicitly encodes a homophily parameter $h$ and class variances. It derives a Bayes-optimal classifier and two node-distinguishability metrics, Probabilistic Bayes Error ($PBE$) and Negative Generalized Jeffreys Divergence ($D_{NGJ}$), to quantify intra- and inter-class ND and analyzes how graph filters, degree distributions, and class variances shape these quantities. The authors demonstrate a mid-homophily pitfall and three operational regimes for filters (full/low/high-pass) and validate that ND, not just homophily, governs GNN advantage on real and synthetic data. They also propose a classifier-based performance metric (CPM) that yields statistically meaningful thresholds (via p-values) for predicting when graph-aware methods outperform graph-agnostic baselines, offering a practical tool beyond traditional homophily metrics. Together, these contributions provide both a theoretical ND framework and a training-free criterion to guide the use of graph structure in node classification.

Abstract

Homophily principle, i.e., nodes with the same labels are more likely to be connected, has been believed to be the main reason for the performance superiority of Graph Neural Networks (GNNs) over Neural Networks on node classification tasks. Recent research suggests that, even in the absence of homophily, the advantage of GNNs still exists as long as nodes from the same class share similar neighborhood patterns. However, this argument only considers intra-class Node Distinguishability (ND) but neglects inter-class ND, which provides incomplete understanding of homophily on GNNs. In this paper, we first demonstrate such deficiency with examples and argue that an ideal situation for ND is to have smaller intra-class ND than inter-class ND. To formulate this idea and study ND deeply, we propose Contextual Stochastic Block Model for Homophily (CSBM-H) and define two metrics, Probabilistic Bayes Error (PBE) and negative generalized Jeffreys divergence, to quantify ND. With the metrics, we visualize and analyze how graph filters, node degree distributions and class variances influence ND, and investigate the combined effect of intra- and inter-class ND. Besides, we discovered the mid-homophily pitfall, which occurs widely in graph datasets. Furthermore, we verified that, in real-work tasks, the superiority of GNNs is indeed closely related to both intra- and inter-class ND regardless of homophily levels. Grounded in this observation, we propose a new hypothesis-testing based performance metric beyond homophily, which is non-linear, feature-based and can provide statistical threshold value for GNNs' the superiority. Experiments indicate that it is significantly more effective than the existing homophily metrics on revealing the advantage and disadvantage of graph-aware modes on both synthetic and benchmark real-world datasets.

Figures (12)

• Figure 1: Example of intra- and inter-class node distinguishability.
• Figure 2: Visualization of CSBM-H $\left(\bm{\mu}_0 = [-1,0],\bm{\mu}_1 =[0,1], \sigma_0^2 = 1,\sigma_1^2 = 2, \right.$$\left. d_0 = 5,d_1 = 5 \right)$
• Figure 3: Comparison of CSBM-H with $\sigma^2_0=1,\sigma^2_1=5$.
• Figure 4: Comparison of CSBM-H with $\sigma^2_0=1.9,\sigma^2_1=2$.
• Figure 5: Comparison of CSBM with different $d_0=5,d_1=25$ setups.

Theorems & Definitions (8)

• Definition 1: Bayes Error Rate
• Definition 2: Generalized Jeffreys Divergence
• proof
• proof
• proof
• proof
• proof
• proof