On the Pitfalls of Analyzing Individual Neurons in Language Models

TL;DR

The paper tackles how to interpret language-model neurons by identifying two pitfalls in neuron-ranking via probes: confounding probe quality with ranking quality and conflating encoded with used information. It compares three ranking methods—Linear, Gaussian, and the probe-free Probeless—across multilingual models (M-BERT and XLM-R) and 156 configurations, revealing that probe strength does not always translate into meaningful rankings of used information. Through intervention-based evaluations (ablation and translation), it shows that Gaussian-based probing can memorize rather than reflect model usage, while Probeless better captures neurons actually used by the model to produce outputs for the target attribute. The findings advocate separating ranking from probing and emphasize evaluating used information to improve interpretability and controllability of language models.

Abstract

While many studies have shown that linguistic information is encoded in hidden word representations, few have studied individual neurons, to show how and in which neurons it is encoded. Among these, the common approach is to use an external probe to rank neurons according to their relevance to some linguistic attribute, and to evaluate the obtained ranking using the same probe that produced it. We show two pitfalls in this methodology: 1. It confounds distinct factors: probe quality and ranking quality. We separate them and draw conclusions on each. 2. It focuses on encoded information, rather than information that is used by the model. We show that these are not the same. We compare two recent ranking methods and a simple one we introduce, and evaluate them with regard to both of these aspects.

Figures (13)

• Figure 1: Layer 7 neurons overlap, using Probeless ranking. Blue squares are above the expected overlap between 2 rankings (Appendix \ref{['overlaps:expected']}), red are below. Major ticks are attributes, minor are languages.
• Figure 2: Layer 2 neurons overlap between every pair of rankings, from 8 randomly selected configs. Gray and black dashed lines show the expected overlap between 2 and 3 random rankings, respectively (Appendix \ref{['overlaps:expected']}).
• Figure 3: Ranking evaluation by probing: The language model creates a word representation (e.g., of the word "was"), which is fed into a neuron-ranking method, to rank its neurons according to their importance for some attribute (e.g., tense). The $k$-highest ranked neurons are fed into a probe, which is trained to predict the attribute.
• Figure 4: Clustering of the three different patterns (\ref{['fig:probing:bert_clusters']}), and an example of each of the patterns (\ref{['fig:probing:bul']}--\ref{['fig:probing:rus']}). Solid lines are top-to-bottom rankings; dashed are random rankings; dotted are bottom-to-top rankings. "X by Y" means classifier X using ranking Y. Some lines are omitted for clarity; complementing figures can be found in Appendix \ref{['appendix:probing']}.
• Figure 5: Ranking evaluation by interventions: The language model creates a word representation (e.g., of the word "was"), which is fed into a neuron-ranking method, to rank its neurons according to their importance for some attribute (e.g., tense). The $k$-highest ranked neurons are modified by an intervention (to a different color in the figure), and the new representation is fed into the rest of the language model's layers, to observe the final model's output.