Language Model Circuits Are Sparse in the Neuron Basis

TL;DR

This paper challenges the view that neuron-level representations in language models are inherently dense by showing that MLP activations can form sparse, faithful circuits comparable to sparse autoencoder bases. It introduces RelP-based attribution to identify causal neuron-level circuits and demonstrates that a circuit of roughly $≈ 10^2$ MLP neurons can control behavior on a subject-verb agreement task, while also recovering latent reasoning steps in a multi-hop capital task. The authors validate the approach on Llama 3.1 8B Instruct and replicate cross-layer transcoders’ findings, showing that neuron-based circuits can match SAE-based results without extra training, and that edge-level RelP attribution yields more faithful connections than Integrated Gradients. They further show these neuron-level circuits generalize to unpaired data and enable steering of outputs, underscoring practical implications for interpretability, controllability, and safety in real-world models.

Abstract

The high-level concepts that a neural network uses to perform computation need not be aligned to individual neurons (Smolensky, 1986). Language model interpretability research has thus turned to techniques such as \textit{sparse autoencoders} (SAEs) to decompose the neuron basis into more interpretable units of model computation, for tasks such as \textit{circuit tracing}. However, not all neuron-based representations are uninterpretable. For the first time, we empirically show that \textbf{MLP neurons are as sparse a feature basis as SAEs}. We use this finding to develop an end-to-end pipeline for circuit tracing on the MLP neuron basis, which locates causal circuitry on a variety of tasks using gradient-based attribution. On a standard subject-verb agreement benchmark (Marks et al., 2025), a circuit of $\approx 10^2$ MLP neurons is enough to control model behaviour. On the multi-hop city $\to$ state $\to$ capital task from Lindsey et al., 2025, we find a circuit in which small sets of neurons encode specific latent reasoning steps (e.g.~`map city to its state'), and can be steered to change the model's output. This work thus advances automated interpretability of language models without additional training costs.

Figures (15)

• Figure 1: Faithfulness and completeness for different choices of representation in the model (residual stream, attention, MLP activations, or MLP outputs) and basis (neurons or SAE) when applying Integrated Gradients, averaged over the 4 SVA tasks with paired data.
• Figure 2: Faithfulness and completeness for Integrated Gradients vs. RelP, for different choices of representation in the model and basis (neurons or SAE), averaged over the 4 SVA tasks with paired data
• Figure 3: Faithfulness and completeness for different choices of representation and basis when applying Integrated Gradients, averaged over the 4 SVA tasks with unpaired data.
• Figure 4: Faithfulness and completeness for Integrated Gradients vs. RelP, for different choices of representation in the model and basis (neurons or SAE), averaged over the 4 SVA tasks with unpaired data
• Figure 5: Faithfulness and completeness for edge-based circuit evaluation on the SVA benchmark. All methods use MLP activations as the neuron basis. Circuits are pruned by removing edges based on attribution scores, with neurons removed when all incoming or outgoing edges are pruned.