Faithful and Stable Neuron Explanations for Trustworthy Mechanistic Interpretability
Ge Yan, Tuomas Oikarinen, Tsui-Wei, Weng
TL;DR
This work provides a theoretical foundation for neuron identification in mechanistic interpretability by treating it as an inverse learning problem. It derives a generalization gap bound and high-probability faithfulness guarantees for concept-neuron similarity across probing datasets, and introduces a bootstrap-based stability analysis plus a BE method to construct concept prediction sets with coverage guarantees. Through synthetic and real-data experiments, the paper validates convergence properties of common similarity metrics and demonstrates how probing data size and concept frequency affect faithfulness and stability. The resulting framework enables more trustworthy, quantifiable explanations of individual neurons, advancing rigorous mechanistic interpretability.
Abstract
Neuron identification is a popular tool in mechanistic interpretability, aiming to uncover the human-interpretable concepts represented by individual neurons in deep networks. While algorithms such as Network Dissection and CLIP-Dissect achieve great empirical success, a rigorous theoretical foundation remains absent, which is crucial to enable trustworthy and reliable explanations. In this work, we observe that neuron identification can be viewed as the inverse process of machine learning, which allows us to derive guarantees for neuron explanations. Based on this insight, we present the first theoretical analysis of two fundamental challenges: (1) Faithfulness: whether the identified concept faithfully represents the neuron's underlying function and (2) Stability: whether the identification results are consistent across probing datasets. We derive generalization bounds for widely used similarity metrics (e.g. accuracy, AUROC, IoU) to guarantee faithfulness, and propose a bootstrap ensemble procedure that quantifies stability along with BE (Bootstrap Explanation) method to generate concept prediction sets with guaranteed coverage probability. Experiments on both synthetic and real data validate our theoretical results and demonstrate the practicality of our method, providing an important step toward trustworthy neuron identification.