Towards Understanding Sensitive and Decisive Patterns in Explainable AI: A Case Study of Model Interpretation in Geometric Deep Learning

TL;DR

This work distinguishes sensitive patterns (model-driven) from decisive patterns (task-driven) in explainable AI for geometric deep learning, and systematically benchmarks post-hoc and self-interpretable interpretability methods across three GDL backbones on four scientific datasets. It finds that post-hoc methods typically align with sensitive patterns but poorly with decisive patterns, while certain self-interpretable methods (notably LRI-induced) align well with decisive patterns and can be more stable. The authors demonstrate an ensemble strategy that combines post-hoc interpretations from multiple trained models to improve the detection of decisive patterns, and show that higher model accuracy tends to improve alignment between patterns. These results provide practical guidance for choosing interpretability approaches based on whether the goal is understanding model sensitivity or uncovering task-driven causal patterns in scientific applications. The work also contributes by extending GNN-focused interpretability methods to GDL and releasing a modular evaluation platform for principled comparisons.

Abstract

The interpretability of machine learning models has gained increasing attention, particularly in scientific domains where high precision and accountability are crucial. This research focuses on distinguishing between two critical data patterns -- sensitive patterns (model-related) and decisive patterns (task-related) -- which are commonly used as model interpretations but often lead to confusion. Specifically, this study compares the effectiveness of two main streams of interpretation methods: post-hoc methods and self-interpretable methods, in detecting these patterns. Recently, geometric deep learning (GDL) has shown superior predictive performance in various scientific applications, creating an urgent need for principled interpretation methods. Therefore, we conduct our study using several representative GDL applications as case studies. We evaluate thirteen interpretation methods applied to three major GDL backbone models, using four scientific datasets to assess how well these methods identify sensitive and decisive patterns. Our findings indicate that post-hoc methods tend to provide interpretations better aligned with sensitive patterns, whereas certain self-interpretable methods exhibit strong and stable performance in detecting decisive patterns. Additionally, our study offers valuable insights into improving the reliability of these interpretation methods. For example, ensembling post-hoc interpretations from multiple models trained on the same task can effectively uncover the task's decisive patterns.

Figures (4)

• Figure 1: Overview of GDL model interpretation and its evaluation: Interpretation in geometric deep learning (GDL) tasks involves identifying a subset of points $C_s$ from the input point cloud $C$. Decisive patterns are a subset of points that inherently dictate the labels of the point cloud, specified by the learning task, and their identification accuracy is measured by the alignment between $C_s$ and the true decisive patterns (Interpretation ROC-AUC). Sensitive patterns, on the other hand, are the subset of most influential points affecting the model’s predictions, as specified by the model itself. The evaluation of the model’s sensitivity involves assessing the changes of its predictions when $C_s$ is either added to or removed from the input (Fidelity AUC).
• Figure 2: An overview of the interpretation methods benchmarked. Our evaluation considers two major categories: post-hoc and self-interpretable methods. Within the post-hoc methods, four sub-categories are further segmented based on the techniques employed. Similarly, the self-interpretable methods are organized into three distinct categories, each also differentiated by the used techniques.
• Figure 3: Decisive-Induced Fidelity AUC of various models for the three backbone models. 50 models were trained for each backbone per dataset.
• Figure 4: Classification ROC-AUC v.s. Interpretation ROC-AUC for both post-hoc and self-interpretable methods on $\operatorname{SynMol}$ and $\operatorname{ActsTrack}$ datasets.