Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking the Effectiveness of Graph Classification Datasets in Benchmarks for Assessing GNNs

Zhengdao Li, Yong Cao, Kefan Shuai, Yiming Miao, Kai Hwang

TL;DR

This work questions whether graph classification benchmarks truly distinguish GNN advancements from simple baselines. It introduces an empirical protocol to fairly compare graph-aware methods against structure- and attribute-only baselines, and defines a dataset effectiveness metric E(D) that accounts for both dataset difficulty and class count. Through 16 real-world datasets, the study shows that many benchmarks exhibit small discriminative gaps, while others reveal meaningful differences in how structure and attributes drive performance; it also demonstrates a correlation-driven approach to generate synthetic datasets for deeper analysis. Finally, the authors show that dataset effectiveness can be predicted from graph properties using lightweight regressors, offering a practical tool for benchmark selection and future roadmapping in graph learning.

Abstract

Graph classification benchmarks, vital for assessing and developing graph neural networks (GNNs), have recently been scrutinized, as simple methods like MLPs have demonstrated comparable performance. This leads to an important question: Do these benchmarks effectively distinguish the advancements of GNNs over other methodologies? If so, how do we quantitatively measure this effectiveness? In response, we first propose an empirical protocol based on a fair benchmarking framework to investigate the performance discrepancy between simple methods and GNNs. We further propose a novel metric to quantify the dataset effectiveness by considering both dataset complexity and model performance. To the best of our knowledge, our work is the first to thoroughly study and provide an explicit definition for dataset effectiveness in the graph learning area. Through testing across 16 real-world datasets, we found our metric to align with existing studies and intuitive assumptions. Finally, we explore the causes behind the low effectiveness of certain datasets by investigating the correlation between intrinsic graph properties and class labels, and we developed a novel technique supporting the correlation-controllable synthetic dataset generation. Our findings shed light on the current understanding of benchmark datasets, and our new platform could fuel the future evolution of graph classification benchmarks.

Rethinking the Effectiveness of Graph Classification Datasets in Benchmarks for Assessing GNNs

TL;DR

This work questions whether graph classification benchmarks truly distinguish GNN advancements from simple baselines. It introduces an empirical protocol to fairly compare graph-aware methods against structure- and attribute-only baselines, and defines a dataset effectiveness metric E(D) that accounts for both dataset difficulty and class count. Through 16 real-world datasets, the study shows that many benchmarks exhibit small discriminative gaps, while others reveal meaningful differences in how structure and attributes drive performance; it also demonstrates a correlation-driven approach to generate synthetic datasets for deeper analysis. Finally, the authors show that dataset effectiveness can be predicted from graph properties using lightweight regressors, offering a practical tool for benchmark selection and future roadmapping in graph learning.

Abstract

Graph classification benchmarks, vital for assessing and developing graph neural networks (GNNs), have recently been scrutinized, as simple methods like MLPs have demonstrated comparable performance. This leads to an important question: Do these benchmarks effectively distinguish the advancements of GNNs over other methodologies? If so, how do we quantitatively measure this effectiveness? In response, we first propose an empirical protocol based on a fair benchmarking framework to investigate the performance discrepancy between simple methods and GNNs. We further propose a novel metric to quantify the dataset effectiveness by considering both dataset complexity and model performance. To the best of our knowledge, our work is the first to thoroughly study and provide an explicit definition for dataset effectiveness in the graph learning area. Through testing across 16 real-world datasets, we found our metric to align with existing studies and intuitive assumptions. Finally, we explore the causes behind the low effectiveness of certain datasets by investigating the correlation between intrinsic graph properties and class labels, and we developed a novel technique supporting the correlation-controllable synthetic dataset generation. Our findings shed light on the current understanding of benchmark datasets, and our new platform could fuel the future evolution of graph classification benchmarks.
Paper Structure (28 sections, 1 theorem, 3 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 28 sections, 1 theorem, 3 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Empirical Studies of Existing Graph Classification Datasets
  3. An Empirical Protocol for Evaluating Dataset Discriminability
  4. Evaluation Framework.
  5. Assessment of performance gap.
  6. Choices of baselines.
  7. Choices of graph-aware methods.
  8. Collection of Diverse Datasets
  9. Bio&Chem.
  10. Social science.
  11. Computer vision (CV).
  12. Observations of Performance Gaps on 16 Real-world Datasets
  13. Experimental setup.
  14. Limitations of Using Performance Gap as Effectiveness Measurement
  15. Quantifying the Effectiveness of Benchmark Datasets
  16. ...and 13 more sections

Key Result

Theorem 1

Given a set of property variables $\{\mathcal{P}_i\}_{i=1}^K$, each $\mathcal{P}_i$ follows a Gaussian distribution $\mathcal{N}(\mu_k, \sigma_k)$ or Uniform distribution $\mathcal{U}(a_k, b_k)$, and given corresponding Pearson correlation coefficients $\{r_i\}_{i=1}^{K}$ with label variable $\mathc where $\sigma_{\mathcal{Y}}$ is any desired standard deviation, and each $n_i$ is mutually independ

Figures (6)

  • Figure 1: The performance gaps on 16 graph classification datasets are categorized into two types: Ineffective (gray) and Effective (red) benchmarks. These are sorted in ascending order based on the size of the performance gap. An empirical threshold of 10% is used for categorization, as observed in the inner box of each figure. This box represents the distribution of the accuracy gap for GCN and GIN.
  • Figure 2: Properties illustration of $\lambda$ and $\mathcal{E}$.
  • Figure 3: Effectiveness using Accuracy metric and AUC-ROC metric in terms of structural type and attributed type.
  • Figure 4: Correlations between graph property sequences and class labels on 9 real-world datasets.
  • Figure 5: Generated $\mathbb{Y}$ with 11 classes by $\mathcal{P}_2, \mathcal{P}_2, \mathcal{P}_3$ following two uniform and one Gaussian distributions with the correlations ${r_1=-0.7, r_2=0.1, r_3=0.7}$ respectively.
  • ...and 1 more figures

Theorems & Definitions (3)

  • Definition 1
  • Definition 2
  • Theorem 1