Identifying Good and Bad Neurons for Task-Level Controllable LLMs

TL;DR

The paper tackles the challenge of interpreting LLMs at the task level by identifying both facilitative and inhibitory neurons, rather than focusing solely on positive contributors. It introduces NeuronLLM, a framework built on functional antagonism, with AQUA to generate robust proxy questions and CNI with ACE scoring for contrastive neuron attribution, plus intervention mechanisms to validate effects. Across four NLP tasks and multiple model families, NeuronLLM outperforms state-of-the-art baselines in task-level control, demonstrates the value of considering both good and bad neurons, and reveals insights on shared versus task-specific neuron roles and their layer distribution. This work advances interpretability and controllability of LLMs by showing how coordinated neuron dynamics underlie task execution and how to manipulate them reliably.

Abstract

Large Language Models have demonstrated remarkable capabilities on multiple-choice question answering benchmarks, but the complex mechanisms underlying their large-scale neurons remain opaque, posing significant challenges for understanding and steering LLMs. While recent studies made progress on identifying responsible neurons for certain abilities, these ability-specific methods are infeasible for task-focused scenarios requiring coordinated use of multiple abilities. Moreover, these approaches focus only on supportive neurons that correlate positively with task completion, while neglecting neurons with other roles-such as inhibitive roles-and misled neuron attribution due to fortuitous behaviors in LLMs (i.e., correctly answer the questions by chance rather than genuine understanding). To address these challenges, we propose NeuronLLM, a novel task-level LLM understanding framework that adopts the biological principle of functional antagonism for LLM neuron identification. The key insight is that task performance is jointly determined by neurons with two opposing roles: good neurons that facilitate task completion and bad neurons that inhibit it. NeuronLLM achieves a holistic modeling of neurons via contrastive learning of good and bad neurons, while leveraging augmented question sets to mitigate the fortuitous behaviors in LLMs. Comprehensive experiments on LLMs of different sizes and families show the superiority of NeuronLLM over existing methods in four NLP tasks, providing new insights into LLM functional organization.

Figures (9)

• Figure 1: Overview of NeuronLLM. It first generate proxy questions with shuffled answer options, on which a cross-entropy-based neuron attribution is devised to identify good and bad neurons for task-level steering of LLMs.
• Figure 2: Cross-task evaluation.
• Figure 3: The example of a AQUA-transformed question, consisting the components introduced in Section \ref{['sec:qatt']}.
• Figure 4: Evaluation of cross-task abilities of the neurons. The vertical axis represents the task data used for evaluation, while the horizontal axis indicates the task-specific neurons identified by NeuronLLM (excluding common neurons across tasks).
• Figure 5: Intuitive explanation for the enhancement vs. degradation asymmetry.