GNN Explanations that do not Explain and How to find Them

TL;DR

This paper reveals a counterintuitive vulnerability in Self-Explainable Graph Neural Networks: explanations can be completely unfaithful to the model’s inference yet yield near-perfect accuracy. It provides theoretical conditions showing how label-encoding via anchor sets can produce degenerate explanations, and demonstrates both malicious manipulation and natural emergence of such explanations. To counter this, the authors introduce Extension Sufficiency Test (EST), a robust, perturbation-spanning faithfulness metric, and an empirical benchmark for evaluating faithfulness metrics. Through extensive experiments across multiple graph tasks, EST consistently identifies unfaithful explanations where prior metrics fail, highlighting the need for auditable, reliable explanations in high-stakes settings. Overall, the work emphasizes auditing and robust verification of explanations to ensure that SE-GNNs reveal genuine reasoning rather than superficial or concealed patterns, especially regarding protected attributes.

Abstract

Explanations provided by Self-explainable Graph Neural Networks (SE-GNNs) are fundamental for understanding the model's inner workings and for identifying potential misuse of sensitive attributes. Although recent works have highlighted that these explanations can be suboptimal and potentially misleading, a characterization of their failure cases is unavailable. In this work, we identify a critical failure of SE-GNN explanations: explanations can be unambiguously unrelated to how the SE-GNNs infer labels. We show that, on the one hand, many SE-GNNs can achieve optimal true risk while producing these degenerate explanations, and on the other, most faithfulness metrics can fail to identify these failure modes. Our empirical analysis reveals that degenerate explanations can be maliciously planted (allowing an attacker to hide the use of sensitive attributes) and can also emerge naturally, highlighting the need for reliable auditing. To address this, we introduce a novel faithfulness metric that reliably marks degenerate explanations as unfaithful, in both malicious and natural settings. Our code is available in the supplemental.

Figures (21)

• Figure 1: SE-GNNs couple an explanation extractor $e$ producing an explanation $R$, and a classifier $g$ using $R$ to infer the prediction. We identify a critical failure case of SE-GNNs, i.e., degenerate explanations -- explanations that encode the label but are unrelated to how the model actually infers it. In this example, green () and violet () nodes appear identically in every graph and form an anchor set $\mathcal{Z}$, hence they have no class-discriminative power by construction (see \ref{['sec:failure-cases']} and \ref{['ex:red-blue']} for details). Yet the explanation extractor $e\xspace$ can exploit them to secretly encode the predicted label while hiding the use of red and blue nodes. Such explanations are highly unfaithful and mislead users by falsely suggesting that green and violet nodes are relevant for the label.
• Figure 2: Ablation study on the budget of perturbations to estimate \ref{['eq:rejection-ratio']} for GSAT on RBGV (cf. \ref{['sec:proposing-new-metric']}). The plot shows that $\mathsf{EST}$ steadily rejects more unfaithful explanations as the budget increases, while other metrics do not substantially improve. $\mathsf{Fid}\hbox{-}$ ($\mathsf{Fid}\hbox{+}$) have constant values, a they do not apply any sampling but just erase the complement (the explanation) once. Note that $\mathsf{EST}$ samples subgraphs of the complement at random, and thus, the probability of sampling duplicates increases as the budget increases. Future work can investigate more complex sampling strategies to avoid such duplicates.
• Figure 3: SE-GNNs can be manipulated to output unfaithful but highly plausible explanations. We train GSAT (left) and SMGNN (right) to maximize the downstream accuracy while also optimizing for plausibility, that is, outputting the digit and its 1-hop neighbor as an explanation. Nonetheless, both models are heavily relying on the color information, as shown by the severe drop in accuracy when the color information is altered (shown in \ref{['tab:cpatchmnist-black-out']}), and by the arbitrary manipulation of the predicted label by swapping colors (shown in \ref{['fig:cpatchmnist-swap-colors']}). The underlying dataset is an extension of MNISTsp, called CPatchMNIST, in which we color the upper-right and bottom-left superpixels with class-discriminative colors that are treated as protected attributes. Details about the dataset and the color coding are in \ref{['app:plausible-but-unfaithful-experiment']}. The plot shows two graph examples of class $1$, where nodes belonging to the explanation are selected based on a threshold value on the relevance score at $0.5$. Numbers above each node represent raw explanation relevance scores. Better seen in digital format.
• Figure 4: Despite GSAT's explanation declaring that only the digit information is used for the graph in \ref{['fig:cpatchmnist']}, swapping the color to that of other classes, following the schema of \ref{['tab:cpatchmnist-encoding']}, results in arbitrary manipulation of the final prediction.
• Figure 5: Explanations provided by the attacked GSAT on RBGV. Numbers above each node represent raw explanation relevance scores, and explanatory nodes are selected based on a $0.5$ threshold. Better seen in digital format.

Theorems & Definitions (13)

• theorem 1
• Example 1
• definition 1: $\mathsf{EST}$
• theorem 1
• proof
• definition 2: Subgraph anchor set
• theorem 2
• proof
• definition 3: Prime Implicant explanation azzolin2025formal
• definition 4: Minimal explanation azzolin2025formal