Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models

TL;DR

This survey reframes Mechanistic Interpretability (MI) from a predominantly observational practice into an actionable discipline by introducing the Locate, Steer, and Improve pipeline. It formalizes core interpretable objects in decoder-only Transformers, categorizes localization methods (e.g., Magnitude Analysis, Causal Attribution, Gradient Detection, Probing, Vocabulary Projection, Circuit Discovery), and details steering techniques (Amplitude Manipulation, Targeted Optimization, Vector Arithmetic) that enable tangible improvements in Alignment, Capability, and Efficiency. The work synthesizes a broad ecosystem of methods into eight application paradigms (Safety, Fairness, Persona, Multilingualism, Knowledge Editing, Reasoning, and Efficiency) and offers a curated repository of 200+ MI papers with method-to-object tagging. It also highlights challenges—scalability, evaluation fidelity, and robustness of interventions—and presents a path toward integrating MI insights into principled model design for safer, more capable, and efficient LLMs.

Abstract

Mechanistic Interpretability (MI) has emerged as a vital approach to demystify the opaque decision-making of Large Language Models (LLMs). However, existing reviews primarily treat MI as an observational science, summarizing analytical insights while lacking a systematic framework for actionable intervention. To bridge this gap, we present a practical survey structured around the pipeline: "Locate, Steer, and Improve." We formally categorize Localizing (diagnosis) and Steering (intervention) methods based on specific Interpretable Objects to establish a rigorous intervention protocol. Furthermore, we demonstrate how this framework enables tangible improvements in Alignment, Capability, and Efficiency, effectively operationalizing MI as an actionable methodology for model optimization. The curated paper list of this work is available at https://github.com/rattlesnakey/Awesome-Actionable-MI-Survey.

Figures (13)

• Figure 1: Overview of the paper structure. We begin by defining the core interpretable objects (§\ref{['sec:core_objects']}) that form the foundation of our analysis. We then introduce a range of methods, ranging from localization (§\ref{['sec:localize_methods']}) to steering(§\ref{['sec:steer_methods']}). Finally, we illustrate how these methods can be applied to improve models (§\ref{['sec:applications']}).
• Figure 2: The schematic of information flow within a standard Transformer block. The residual stream ($\mathbf{x}_l$) serves as the backbone, while MHA and FFN act as additive branches that read from and write to this stream. Based on the figure from ferrando2024primer.
• Figure 3: The framework of Sparse Autoencoders (SAEs). The SAE acts as an independent module attached to a frozen LLM, expanding dense representations into a sparse, overcomplete set of interpretable features via an encoder-decoder architecture. Based on the figure from shu-etal-2025-survey.
• Figure 4: Localization via Magnitude Analysis.(a) Discovery of SAE reasoning features galichin2025have: SAE features are scored using ReasonScore, which aggregates activation magnitude and frequency during reasoning steps, isolating sparse features that encode cognitive behaviors like uncertainty or reflection. (b) Identification of Style-Specific Neurons lai-etal-2024-style: Neurons are ranked by their average activation magnitude on style-specific corpora, revealing clusters that selectively activate for distinct linguistic styles.
• Figure 5: Overview of Causal Tracing. The method identifies critical internal states by creating a corrupted run (noising the subject "Space Needle") and systematically restoring clean states to see which ones recover the prediction "Seattle". The heatmap results reveal that factual information is processed in early MLP layers at the subject position and later transferred to the final token via attention. Based on the figure from meng2022ccs.