Auxiliary Metrics Help Decoding Skill Neurons in the Wild

TL;DR

The paper tackles interpretability in LLMs by identifying skill-specific neurons through correlations between FFN activations on a trained soft prompt and auxiliary metrics. It extends prior skill-neuron work to multi-skill settings and uses soft-prompt tuning with frozen models to avoid altering core weights, enabling broad applicability without manual token aggregation. The method is validated on open-ended generation (Skill-Mix), natural language inference (HANS), and arithmetic task discovery (BigBench), revealing sparse neurons that align with known skills and uncovering a previously unknown arithmetic shortcut. These findings advance mechanistic understanding of LLMs and point to routes for causal intervention and safer deployment.

Abstract

Large language models (LLMs) exhibit remarkable capabilities across a wide range of tasks, yet their internal mechanisms remain largely opaque. In this paper, we introduce a simple, lightweight, and broadly applicable method with a focus on isolating neurons that encode specific skills. Building upon prior work that identified "skill neurons" via soft prompt training on classification tasks, our approach extends the analysis to complex scenarios involving multiple skills. We correlate neuron activations with auxiliary metrics -- such as external labels and the model's own confidence score -- thereby uncovering interpretable and task-specific behaviors without the need for manual token aggregation. We empirically validate our method on tasks spanning open-ended text generation and natural language inference, demonstrating its ability to detect neurons that not only drive known skills but also reveal previously unidentified shortcuts in arithmetic reasoning on BigBench.

Figures (7)

• Figure 1: Overview of Our Methodology We calculate the correlation between feedforward-layer neuron activations on a trained soft prompt and an auxiliary metric to identify skill-related neurons.
• Figure 2: Distribution of activations of the neuron with highest absolute correlation by skill (Skill-Mix). The distribution is interpolated by Kernel density estimation (KDE) based on the empirical distribution of activation over validation set.
• Figure 3: Distribution of activations of the neuron with highest absolute correlation on data with three different heuristics on HANS.
• Figure 4: Distribution of correlation values between each neuron's activation and the HANS heuristic label. The red dashed line indicates the top-10 correlation score threshold (0.43).
• Figure 5: Distribution of activations of the neuron with highest absolute correlation on the data of the Arithmetic task. The shortcut indicates that the correct answer for a multiplication question can be determined solely by the last digit, a pattern automatically discovered by our algorithm.