Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations

TL;DR

A pipeline is presented that bridges circuit-level analysis and natural language explanations by identifying causally important attention heads via activation patching, generating explanations using both template-based and LLM-based methods, and evaluating faithfulness using ERASER-style metrics adapted for circuit-level attribution.

Abstract

Mechanistic interpretability identifies internal circuits responsible for model behaviors, yet translating these findings into human-understandable explanations remains an open problem. We present a pipeline that bridges circuit-level analysis and natural language explanations by (i) identifying causally important attention heads via activation patching, (ii) generating explanations using both template-based and LLM-based methods, and (iii) evaluating faithfulness using ERASER-style metrics adapted for circuit-level attribution. We evaluate on the Indirect Object Identification (IOI) task in GPT-2 Small (124M parameters), identifying six attention heads accounting for 61.4% of the logit difference. Our circuit-based explanations achieve 100% sufficiency but only 22% comprehensiveness, revealing distributed backup mechanisms. LLM-generated explanations outperform template baselines by 64% on quality metrics. We find no correlation (r = 0.009) between model confidence and explanation faithfulness, and identify three failure categories explaining when explanations diverge from mechanisms.

Figures (7)

• Figure 1: Pipeline overview. We identify causally important heads via activation patching, generate NL explanations using templates or LLMs, and evaluate faithfulness using adapted ERASER metrics (sufficiency, comprehensiveness, F1).
• Figure 2: IOI circuit identification via activation patching. Each cell shows a head's causal contribution. L9H9 (Name Mover) shows highest importance at 17.4%.
• Figure 3: ERASER metric comparison. Our circuit-based method achieves 100% sufficiency and outperforms the attention baseline by 75% on F1 score.
• Figure 4: Explanation quality comparison. LLM-generated explanations achieve 99% quality vs. 60% for templates (+66%).
• Figure 5: Comprehensiveness distribution across 50 prompts. 34% of cases show low comprehensiveness ($<$15%), indicating explanation-mechanism divergence.