Capability Localization: Capabilities Can be Localized rather than Individual Knowledge

TL;DR

The paper interrogates whether individual knowledge in large language models can be localized to specific parameters, presenting fidelity and reliability evidence that existing localization methods are unreliable. It then shows that while individual knowledge cannot be localized, data commonalities can be captured by parameters through decoupling experiments, leading to the proposal of Commonality Neuron Localization (CNL). CNL locates commonality neurons with high cross-dataset overlap (e.g., $96.42\%$ overlap and $95.95\%$ IoU on GSM8K) and demonstrates that these capability neurons can enhance performance across datasets when activated or fine-tuned. The findings establish a shift from localizing individual knowledge to localizing data capabilities, offering a scalable, cross-task approach with demonstrated generalization and practical implications for parameter-efficient localization.

Abstract

Large scale language models have achieved superior performance in tasks related to natural language processing, however, it is still unclear how model parameters affect performance improvement. Previous studies assumed that individual knowledge is stored in local parameters, and the storage form of individual knowledge is dispersed parameters, parameter layers, or parameter chains, which are not unified. We found through fidelity and reliability evaluation experiments that individual knowledge cannot be localized. Afterwards, we constructed a dataset for decoupling experiments and discovered the potential for localizing data commonalities. To further reveal this phenomenon, this paper proposes a Commonality Neuron Localization (CNL) method, which successfully locates commonality neurons and achieves a neuron overlap rate of 96.42% on the GSM8K dataset. Finally, we have demonstrated through cross data experiments that commonality neurons are a collection of capability neurons that possess the capability to enhance performance. Our code is available at https://github.com/nlpkeg/Capability-Neuron-Localization.

Figures (8)

• Figure 1: Overall framework diagram of fidelity and reliability evaluation experiments. Experiment A represents the visualization results of samples with the same semantic localization, while experiment B represents the impact of operating localization neurons on performance
• Figure 2: The performance of editing at different layers, the horizontal axis is layer number.
• Figure 3: Overall framework diagram of decoupling experiments. The upper half represents the overlapping neurons for obtaining the localization of subsample1 and subsample2, while the lower half represents the composition of the subsample, consisting of the main and replaceable parts.
• Figure 4: Visualisation of commonality neurons. For ease of observation, we choose the neuron with the highest absolute value among 100 neighboring neurons. The horizontal axis shows neuron IDs, the vertical axis shows model layer IDs, and darker squares indicate more prominent located neurons.
• Figure 5: Effects of capacity neurons on other dataset. The vertical axis represents the enhanced or erased dataset, and the horizontal axis represents the tested dataset. The number represents the performance difference between the enhanced or zeroed model and the base model.