Emergent Specialization: Rare Token Neurons in Language Models

TL;DR

This work tackles the problem of how large language models represent and predict rare tokens by identifying a small set of rare token neurons in the final MLP that disproportionately influence infrequent token predictions. It combines ablation-based neuron influence measurements, phase-transition analysis of neuron contributions, and activation-space geometry to reveal a dynamic three-phase organization—plateau, power-law, and rapid decay—alongside coordinated co-activation among rare-token neurons. The study connects these functional patterns to heavy-tailed weight distributions via HT-SR theory, showing consistently lower $\alpha_{\text{Hill}}$ for rare-token neurons and constructing a spectral-structural explanation for emergent specialization. These findings provide a mechanistic view of internal specialization in LLMs, offering avenues for data-efficient training and principled domain adaptation through targeted subnetworks and spectral-informed regularization.

Abstract

Large language models struggle with representing and generating rare tokens despite their importance in specialized domains. In this study, we identify neuron structures with exceptionally strong influence on language model's prediction of rare tokens, termed as rare token neurons, and investigate the mechanism for their emergence and behavior. These neurons exhibit a characteristic three-phase organization (plateau, power-law, and rapid decay) that emerges dynamically during training, evolving from a homogeneous initial state to a functionally differentiated architecture. In the activation space, rare token neurons form a coordinated subnetwork that selectively co-activates while avoiding co-activation with other neurons. This functional specialization potentially correlates with the development of heavy-tailed weight distributions, suggesting a statistical mechanical basis for emergent specialization.

Figures (4)

• Figure 1: Neuron influence distribution and phase distinctions. The $\log\Delta$loss-$\log$rank relation shown in figure (b) reveals a three-phase structure: highly-influential plateau (blue) consisting of 1.7% of neurons, mid-rank power-law phase (green) with 10.(% of neurons, and low-rank rapid decay phase (red) with the remaining 87.4% of neurons.
• Figure 2: As shown in (a), in $\log-\log$ coordinates, the green line indicates the power law. For the least influential neurons on the rightmost, power-law fails for a rapid drop of influence. While for the most influential neurons on the leftmost, the power-law fails due to an emergence of additional bias, while the slope remains as the power-law regime. In (b) we illustrate the dynamical deviation of top-ranked neurons from power-law predictions. At early steps of training (blue), the bias is close to 0 across log rank, indicating the power-law describes the whole rank regime $\log\text{rank}\in (0,4)$. While as training proceeds to later steps (green), top-ranked neurons deviate from power law prediction and from a plateau around the regime $\log\text{rank}\in (0,1.5)$. This plateau becomes more evident at late training steps (red), where neurons ranked $\log\text{rank}\in (0,3)$ all deviate from power-law prediction significantly.
• Figure 3: Parallel emergence of functional specialization and statistical heavy-tailedness during model training. (a) The slope distribution evolves to form distinct neuronal regimes at higher training steps. (b) Specialized neurons develop increasingly heavy-tailed weight distributions compared to random neurons, hinting a link between functional differentiation and statistical properties.
• Figure 4: Distribution of neuron influence slopes for suppress neurons across the GPT-2 model family, showing characteristic three-phase organization emerging during training.

Theorems & Definitions (2)

• Conjecture 5.1: Dual-Regime Organization
• Conjecture 5.2: Parallel Mechanism Conjecture