Hyperbolic Benchmarking Unveils Network Topology-Feature Relationship in GNN Performance

TL;DR

The paper tackles the challenge of fairly benchmarking GNNs across graphs with diverse topology and feature couplings. It proposes HypNF, a benchmarking framework built on the $S^1/H^2$ hyperbolic soft configuration model and its bipartite extension to generate synthetic graphs with tunable degree distributions, clustering, homophily, and topology-feature alignment. Empirically, stronger topology-feature coupling and hyperbolic embeddings yield advantages, especially for link prediction, while simple feature-based baselines can compete in node classification under certain conditions. By providing an open-source, controllable data generator, the work enables standardized model comparisons and practical guidance for model selection in real-world datasets.

Abstract

Graph Neural Networks (GNNs) have excelled in predicting graph properties in various applications ranging from identifying trends in social networks to drug discovery and malware detection. With the abundance of new architectures and increased complexity, GNNs are becoming highly specialized when tested on a few well-known datasets. However, how the performance of GNNs depends on the topological and features properties of graphs is still an open question. In this work, we introduce a comprehensive benchmarking framework for graph machine learning, focusing on the performance of GNNs across varied network structures. Utilizing the geometric soft configuration model in hyperbolic space, we generate synthetic networks with realistic topological properties and node feature vectors. This approach enables us to assess the impact of network properties, such as topology-feature correlation, degree distributions, local density of triangles (or clustering), and homophily, on the effectiveness of different GNN architectures. Our results highlight the dependency of model performance on the interplay between network structure and node features, providing insights for model selection in various scenarios. This study contributes to the field by offering a versatile tool for evaluating GNNs, thereby assisting in developing and selecting suitable models based on specific data characteristics.

Figures (14)

• Figure 1: (a) Representation of HypNF benchmarking framework to generate rich graph-structured data. (b) Hyperbolic representation of a synthetic network with $N_n = 2000$ nodes represented as circles, where colors indicate their labels, and $N_f = 200$ features depicted as purple squares. The size of the symbols is proportional to the logarithms of the degrees of nodes and features. The parameters for the $\mathbb{S}^1$ model are $\beta = 3$, $\gamma = 3.5$, and $\langle k \rangle = 30$, and for the bipartite-$\mathbb{S}^1$ model, $\beta_b = 3$, $\gamma_n = 3.5$, $\gamma_f = 2.1$, and $\langle k_n \rangle = 3$. Only edges with an effective distance $\chi < 1$ are depicted in the figure.
• Figure 2: The impact of topology-feature correlation controlled by $\beta$ and $\beta_b$ on the performance of graph machine learning models in two tasks: (a) node classification and (b) link prediction. We set $\mathcal{N}_L=6$ and $\alpha=10$ for the NC task. The box ranges from the first quartile to the third quartile. A horizontal line goes through the box at the median. The whiskers go from each quartile to the minimum or maximum. The results are averaged over all other parameters.
• Figure 3: Impact of each individual parameter on the performance of NC and LP. In the case of NC, we set $\mathcal{N}_L=6$ and $\alpha=10$.
• Figure 4: Aggregated results for (a) link prediction and (b) node classification tasks. The annotations denote the statistical significance levels derived from the Mann-Whitney U test, a non-parametric test suitable for comparing two independent samples, especially when the data distribution is not assumed to be normal. This test evaluates if there is a statistically significant difference in the median performance scores between two different models. The label $\textit{ns}$ signifies p-value $\le 1$, $\textbf{****}$ corresponds to p-value $\le 10^{-4}$.
• Figure 5: Topological properties of Cora dataset and its synthetic counterpart generated by HypNF. (a)-(c) Complementary cumulative degree distribution of nodes in $\mathcal{G}_n$ and those of nodes and features in $\mathcal{G}_{n,f}$. (d)-(f) clustering spectrum of nodes as a function of node degrees in $\mathcal{G}_n$ and bipartite clustering sepctrum of nodes and features as a function of nodes and features degrees in $\mathcal{G}_{n,f}$. (g)-(i) average nearest neighbors degree function in $\mathcal{G}_n$ and $\mathcal{G}_{n,f}$. The clustering coefficient, $c$, for each node in $\mathcal{G}_n$ is calculated as the fraction of connected pairs among its neighboring nodes relative to the total possible pairs of neighbors. To compute the bipartite clustering, $c_b$, for a given node, each pair of its neighboring features is considered connected if they share at least one common node, apart from the node in question. The clustering coefficient is then computed similarly to the clustering coefficient in a unipartite network. This definition also applies to the bipartite clustering coefficient of features. Finally, $k_{nn}$ in both $\mathcal{G}_n$ and $\mathcal{G}_{n,f}$ quantifies the average degree of the neighbors for a given node or feature. Exponential binning is used in the computation $\overline{c}$, $\overline{c}_b$ and $\overline{k}_{nn}$. The blue shaded region indicates the two-$\sigma$ intervals around the mean, derived from 100 realizations of HypNF.