A method for the systematic generation of graph XAI benchmarks via Weisfeiler-Leman coloring

TL;DR

The paper tackles the problem of unreliable graph-XAI benchmarking by introducing a WL coloring–based method to automatically extract ground-truth motifs from real-world graph classification data. This yields the OpenGraphXAI benchmark suite (15 datasets from molecular graphs) and a codebase to generate 2000+ additional benchmarks, with GT explanations aligned to WL expressiveness to ensure learnability by GNNs. The authors demonstrate the utility of the benchmarks by evaluating multiple graph explainers, finding that CAM often provides the most plausible explanations, and they emphasize the importance of large, diverse benchmarks for robust statistical conclusions. Overall, the approach offers a scalable, WL-consistent framework for assessing graph explainability, potentially accelerating progress in graph-XAI research and enabling more trustworthy graph-based decisions in practice.

Abstract

Graph neural networks have become the de facto model for learning from structured data. However, the decision-making process of GNNs remains opaque to the end user, which undermines their use in safety-critical applications. Several explainable AI techniques for graphs have been developed to address this major issue. Focusing on graph classification, these explainers identify subgraph motifs that explain predictions. Therefore, a robust benchmarking of graph explainers is required to ensure that the produced explanations are of high quality, i.e., aligned with the GNN's decision process. However, current graph-XAI benchmarks are limited to simplistic synthetic datasets or a few real-world tasks curated by domain experts, hindering rigorous and reproducible evaluation, and consequently stalling progress in the field. To overcome these limitations, we propose a method to automate the construction of graph XAI benchmarks from generic graph classification datasets. Our approach leverages the Weisfeiler-Leman color refinement algorithm to efficiently perform approximate subgraph matching and mine class-discriminating motifs, which serve as proxy ground-truth class explanations. At the same time, we ensure that these motifs can be learned by GNNs because their discriminating power aligns with WL expressiveness. This work also introduces the OpenGraphXAI benchmark suite, which consists of 15 ready-made graph-XAI datasets derived by applying our method to real-world molecular classification datasets. The suite is available to the public along with a codebase to generate over 2,000 additional graph-XAI benchmarks. Finally, we present a use case that illustrates how the suite can be used to assess the effectiveness of a selection of popular graph explainers, demonstrating the critical role of a sufficiently large benchmark collection for improving the significance of experimental results.

Figures (8)

• Figure 1: The schematics of message-passing (MP) in GNNs. The representation $\mathbf{h}_v^{(2)}$ of node $v$ at layer $\ell = 2$ is obtained by aggregating and combining the representations $\left\{\mathbf{h}_u^{(1)} : u \in \mathcal{N}_v\right\}$, which in turn are obtained by the same mechanism. After the two MP layers $\text{MP}_1$ and $\text{MP}_2$, all nodes within $2$ hops have contributed to the context of node $v$, that is $\mathop{\mathrm{\mathcal{S}}}\nolimits_\textsf{G}^{(2)}(v)$.
• Figure 2: The schematics of WL coloring. The blue color assigned to the node highlighted in the rightmost graph is obtained by hashing the color of its immediate neighbors at the previous iteration (leftmost graph). Each WL color identifies a unique unfolding tree (right).
• Figure 3: We show the mechanics of the proposed algorithm for automatically generating a GT-annotated benchmark from any graph classification dataset through an exemplary molecule represented as a graph. WL coloring is used to mine the set of sub-graph motifs available for the construction of our XAI benchmarks. A motif corresponding to the WL label $\textsf{m} \in \mathcal{C}_2$ has been chosen. The explanation mask is constituted by the $2$-radius ego-graphs of nodes with WL color $\textsf{m}$. In this example, the mined motif can be traced back to the fragment [CH3][CH2][CH2].
• Figure 4: Explanation masks computed by the tested explainers on a test graph from the mike dataset. Only the top 9 nodes (GT dimension) are highlighted for each mask.
• Figure 5: Statistical significance of the explainer ranking as $p$-value of the Friedman test Demsar2006, varying the number of benchmarks considered. At least $7$ benchmarks are required for $p < 0.01$ significant results. OpenGraphXAI benchmarks allow $p \approx 10^{-8}$, or $>5\sigma$ significance.