Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeuronScope: A Multi-Agent Framework for Explaining Polysemantic Neurons in Language Models

Weiqi Liu, Yongliang Miao, Haiyan Zhao, Yanguang Liu, Mengnan Du

TL;DR

Polysemantic neurons in large language models pose interpretability challenges. NeuronScope reframes neuron explanations as iterative, activation-guided refinements, modeling a neuron with activation $a_n(x)$ and high-activation data $\\mathcal{D}$ as a mixture $\\mathcal{D}=\\{\\mathcal{D}_k\\}_{k=1}^K$ and maximizing $Score(E,\\mathcal{D}_k)$ with $Score(E,\\mathcal{D})=\\mathrm{Corr}(\\hat{\\mathbf{a}}_n(E;\\mathcal{D}),\\mathbf{a}_n(\\mathcal{D}))$. The pipeline comprises Hypothesis Drafting to produce $H_{\\text{raw}}$, Decomposition and Clustering to form atomic components $\\{c_i\\}$ and semantic clusters $\\{S_j\\}$, and Evolutionary Refinement to optimize $E$ per cluster over a fixed budget (often five iterations). Across three open-source LLMs, NeuronScope improves activation-correlation scores and reveals coherent semantic groups, demonstrating meaningful polysemantic disentanglement with practical interpretability benefits. The work advances mechanistic interpretability by enabling faithful, multi-faceted neuron explanations and provides guidelines for iterative, activation-aware refinement, though it notes computational overhead and reliance on activation-sample quality as future considerations.

Abstract

Neuron-level interpretation in large language models (LLMs) is fundamentally challenged by widespread polysemanticity, where individual neurons respond to multiple distinct semantic concepts. Existing single-pass interpretation methods struggle to faithfully capture such multi-concept behavior. In this work, we propose NeuronScope, a multi-agent framework that reformulates neuron interpretation as an iterative, activation-guided process. NeuronScope explicitly deconstructs neuron activations into atomic semantic components, clusters them into distinct semantic modes, and iteratively refines each explanation using neuron activation feedback. Experiments demonstrate that NeuronScope uncovers hidden polysemanticity and produces explanations with significantly higher activation correlation compared to single-pass baselines.

NeuronScope: A Multi-Agent Framework for Explaining Polysemantic Neurons in Language Models

TL;DR

Polysemantic neurons in large language models pose interpretability challenges. NeuronScope reframes neuron explanations as iterative, activation-guided refinements, modeling a neuron with activation and high-activation data as a mixture and maximizing with . The pipeline comprises Hypothesis Drafting to produce , Decomposition and Clustering to form atomic components and semantic clusters , and Evolutionary Refinement to optimize per cluster over a fixed budget (often five iterations). Across three open-source LLMs, NeuronScope improves activation-correlation scores and reveals coherent semantic groups, demonstrating meaningful polysemantic disentanglement with practical interpretability benefits. The work advances mechanistic interpretability by enabling faithful, multi-faceted neuron explanations and provides guidelines for iterative, activation-aware refinement, though it notes computational overhead and reliance on activation-sample quality as future considerations.

Abstract

Neuron-level interpretation in large language models (LLMs) is fundamentally challenged by widespread polysemanticity, where individual neurons respond to multiple distinct semantic concepts. Existing single-pass interpretation methods struggle to faithfully capture such multi-concept behavior. In this work, we propose NeuronScope, a multi-agent framework that reformulates neuron interpretation as an iterative, activation-guided process. NeuronScope explicitly deconstructs neuron activations into atomic semantic components, clusters them into distinct semantic modes, and iteratively refines each explanation using neuron activation feedback. Experiments demonstrate that NeuronScope uncovers hidden polysemanticity and produces explanations with significantly higher activation correlation compared to single-pass baselines.
Paper Structure (21 sections, 4 equations, 3 figures, 3 tables)

This paper contains 21 sections, 4 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem Formulation
  4. Explanation Drafting
  5. Semantic Decomposition and Clustering
  6. Evolutionary Explanation Refinement
  7. Experiments
  8. Experimental Setup
  9. Experimental Results
  10. Convergence Analysis of Refinement
  11. Clustering Forms Semantic Groups
  12. Related Work
  13. Conclusions
  14. Related Work
  15. Implementation Details
  16. ...and 6 more sections

Figures (3)

  • Figure 1: The architecture of NeuronScope. (1) Initial Explanation, where a Hypothesis Agent generates broad raw explanations from high-activation corpus exemplars, prioritizing recall; (2) Decomposition and Clustering, where a Decomposition Agent breaks composite hypotheses into atomic components which are then grouped into distinct semantic modes by a Clustering Agent ; and (3) Evolutionary Explanation Refinement, where a Refinement Agent treats explanation generation as an optimization problem, iteratively updating descriptions based on activation feedback to maximize precision.
  • Figure 2: a. Iterative Score Convergence Across Neurons and Layers. b. Clustering forms semantic groups.
  • Figure 3: Semantic mixing in original neuron explanations. Different colors highlights distinct semantic categories, illustrating semantic impurity before separation.