Revisiting the Necessity of Graph Learning and Common Graph Benchmarks

TL;DR

The paper interrogates the assumed necessity of graph structure in graph learning benchmarks by showing that a carefully tuned MLP operating on node features can rival or exceed graph-based methods on several canonical datasets. It introduces a targeted feature study and a synthetic benchmark family (Watts-Strogatz-based WS1000 and WS1000_gamma) to isolate when graph information is truly needed. The authors also propose alternative real-world benchmarks where graph structure yields tangible gains, and discuss benchmarking practices, limitations, and directions for more robust evaluation. Overall, the work urges caution in interpreting graph-learning progress and provides concrete tools to better quantify when graphs contribute meaningfully to predictive performance.

Abstract

Graph machine learning has enjoyed a meteoric rise in popularity since the introduction of deep learning in graph contexts. This is no surprise due to the ubiquity of graph data in large scale industrial settings. Tacitly assumed in all graph learning tasks is the separation of the graph structure and node features: node features strictly encode individual data while the graph structure consists only of pairwise interactions. The driving belief is that node features are (by themselves) insufficient for these tasks, so benchmark performance accurately reflects improvements in graph learning. In our paper, we challenge this orthodoxy by showing that, surprisingly, node features are oftentimes more-than-sufficient for many common graph benchmarks, breaking this critical assumption. When comparing against a well-tuned feature-only MLP baseline on seven of the most commonly used graph learning datasets, one gains little benefit from using graph structure on five datasets. We posit that these datasets do not benefit considerably from graph learning because the features themselves already contain enough graph information to obviate or substantially reduce the need for the graph. To illustrate this point, we perform a feature study on these datasets and show how the features are responsible for closing the gap between MLP and graph-method performance. Further, in service of introducing better empirical measures of progress for graph neural networks, we present a challenging parametric family of principled synthetic datasets that necessitate graph information for nontrivial performance. Lastly, we section out a subset of real-world datasets that are not trivially solved by an MLP and hence serve as reasonable benchmarks for graph neural networks.

Figures (3)

• Figure 1: We conduct a feature study on the Amazon Computers and Cora datasets by synthesizing datasets with an increasing number of features. These datasets comprise the ticks along the "Features" axis. On each dataset, we thoroughly tuned the MLP and GCN. Means over $5$ trials are reported, and the shaded region indicates one standard deviation. As is clearly visible, there is initially a large gap between the MLP and GCN in both subfigures, yet the gap closes considerably for Amazon Computers as the number of features increases, whereas the same is not true to the same extent for Cora. This indicates that graph information "leaks" via the features for Amazon Computers and explains why in some graph learning contexts the graph is unnecessary to obtain good performance.
• Figure 2: Introducing parental dependence in node features very quickly leads the MLP to improve on link prediction. Each data point is obtained by tuning the MLP on the relevant synthetic $WS1000_\gamma$ dataset. The average of $5$ trials is reported and the error region specifies one standard deviation.
• Figure 3: We conduct feature studies on the Amazon Photo, Pubmed, Coauthor Physics, Coauthor CS, and CiteSeer datasets by synthesizing datasets with an increasing number of features. These datasets comprise the ticks along the "Features" axis. On each dataset, we thoroughly tuned the MLP and GCN. Means over $5$ trials are reported, and the shaded region indicates one standard deviation. As is clearly visible, there is a gap between the MLP and GCN in all subfigures, yet the gap closes considerably as the number of features increases. This indicates that graph information "leaks" via the features and explains why in some graph learning contexts the graph is unnecessary to obtain good performance. Somewhat surprisingly, we even see the MLP match and outperform the GCN in some cases involving Coauthor Physics and Coauthor CS.