Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Revealing Combinatorial Reasoning of GNNs via Graph Concept Bottleneck Layer

Yue Niu, Zhaokai Sun, Jiayi Yang, Xiaofeng Cao, Rui Fan, Xin Sun, Hanli Wang, Wei Ye

TL;DR

A graph concept bottleneck layer that can be integrated into any GNN architectures to guide them to predict the selected discriminative global graph concepts and enforces the combinatorial reasoning of GNNs'predictions to fit the soft logical rule over graph concepts and thus can quantify the contribution of each concept.

Abstract

Despite their success in various domains, the growing dependence on GNNs raises a critical concern about the nature of the combinatorial reasoning underlying their predictions, which is often hidden within their black-box architectures. Addressing this challenge requires understanding how GNNs translate topological patterns into logical rules. However, current works only uncover the hard logical rules over graph concepts, which cannot quantify the contribution of each concept to prediction. Moreover, they are post-hoc interpretable methods that generate explanations after model training and may not accurately reflect the true combinatorial reasoning of GNNs, since they approximate it with a surrogate. In this work, we develop a graph concept bottleneck layer that can be integrated into any GNN architectures to guide them to predict the selected discriminative global graph concepts. The predicted concept scores are further projected to class labels by a sparse linear layer. It enforces the combinatorial reasoning of GNNs' predictions to fit the soft logical rule over graph concepts and thus can quantify the contribution of each concept. To further improve the quality of the concept bottleneck, we treat concepts as "graph words" and graphs as "graph sentences", and leverage language models to learn graph concept embeddings. Extensive experiments on multiple datasets show that our method GCBMs achieve state-of-the-art performance both in classification and interpretability.

Revealing Combinatorial Reasoning of GNNs via Graph Concept Bottleneck Layer

TL;DR

A graph concept bottleneck layer that can be integrated into any GNN architectures to guide them to predict the selected discriminative global graph concepts and enforces the combinatorial reasoning of GNNs'predictions to fit the soft logical rule over graph concepts and thus can quantify the contribution of each concept.

Abstract

Despite their success in various domains, the growing dependence on GNNs raises a critical concern about the nature of the combinatorial reasoning underlying their predictions, which is often hidden within their black-box architectures. Addressing this challenge requires understanding how GNNs translate topological patterns into logical rules. However, current works only uncover the hard logical rules over graph concepts, which cannot quantify the contribution of each concept to prediction. Moreover, they are post-hoc interpretable methods that generate explanations after model training and may not accurately reflect the true combinatorial reasoning of GNNs, since they approximate it with a surrogate. In this work, we develop a graph concept bottleneck layer that can be integrated into any GNN architectures to guide them to predict the selected discriminative global graph concepts. The predicted concept scores are further projected to class labels by a sparse linear layer. It enforces the combinatorial reasoning of GNNs' predictions to fit the soft logical rule over graph concepts and thus can quantify the contribution of each concept. To further improve the quality of the concept bottleneck, we treat concepts as "graph words" and graphs as "graph sentences", and leverage language models to learn graph concept embeddings. Extensive experiments on multiple datasets show that our method GCBMs achieve state-of-the-art performance both in classification and interpretability.
Paper Structure (45 sections, 7 theorems, 49 equations, 7 figures, 10 tables)

This paper contains 45 sections, 7 theorems, 49 equations, 7 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Interpretable GNNs
  4. Concept Bottleneck Models
  5. Preliminaries
  6. Method: GCBMs
  7. Constructing Graph Concept Bottleneck Layer
  8. Graph Concept Embedding
  9. Experimental Evaluation
  10. Experimental Setup
  11. Experimental Results and Analysis
  12. Graph Classification Performance
  13. Interpretability Evaluation
  14. Human Evaluation
  15. Quantitative Assessment
  16. ...and 30 more sections

Key Result

Lemma 4.4

After optimization with the concept loss $\mathcal{L}_{c}(\mathbf{c}, \Phi(\mathbf{h}))$, the mutual KL divergence between predicted concepts $\hat{\mathbf{c}}$ and concept labels $\mathbf{c}$ is bounded: where $T_m \in \mathcal{V}$ and $k \in \{0,1\}$. For simplicity, $T_m$ is omitted in subsequent derivations. $\epsilon \ll \tau$ is a small alignment error, ensuring the predicted concepts $\hat

Figures (7)

  • Figure 1: Overview of the GCBMs construction pipeline.
  • Figure 2: On the left side is a graph, and on the right side is its WL-subtree of height 2 rooted at the node with label 1.
  • Figure 3: Distributions of the WL-subtrees ($\text{height} \leq 2$) in graph datasets MUTAG, COX2, BZR, and DHFR. The $x$-axis denotes WL-subtree rank (i.e., the number of occurrences of a WL-subtree), and the $y$-axis denotes the number of WL-subtrees with the same rank. These distributions approximate a power law.
  • Figure 4: The sankey diagram of the classification layer weights of GCBM on the DHFR dataset. The width of the connection between each concept and class corresponds to the exponent of the absolute value of the learned concept weight. To highlight core concepts, only the top 8 concepts with the largest absolute values of weights in each class are shown.
  • Figure 5: Concept indices and their corresponding specific functional groups in molecules from the DHFR dataset.
  • ...and 2 more figures

Theorems & Definitions (12)

  • Definition 3.1: Graph Concept Bottleneck Layer
  • Lemma 4.4: Strong Concept Alignment Constraint
  • Lemma 4.5: Discriminability Preservation Constraint
  • Proposition 4.6: GCBM Discriminability Lower Bound
  • Proposition 4.7: Black-Box Model Discriminability Degradation
  • proof
  • Lemma 4.8: KL Divergence for Normal Distributions with Equal Variance
  • proof
  • Proposition 4.9: GCBM Mean Difference Superiority
  • Theorem 4.10: GCBM AUC Superiority in Class-Imbalanced Graph Classification
  • ...and 2 more