CRANE: Causal Relevance Analysis of Language-Specific Neurons in Multilingual Large Language Models

TL;DR

CRANE introduces a causal relevance-based framework to identify language-specific neuron contributions in multilingual LLMs. By combining layer-wise relevance propagation with kurtosis-based language contrasts and targeted neuron masking, CRANE defines language specificity in terms of functional necessity and provides LangSpec-F1 as a measure of language-selective degradation $LangSpec ext{-}F1$. Across English, Chinese, and Vietnamese, CRANE finds asymmetric, non-exclusive language specialization: target-language neurons disproportionately affect the target language while preserving others, and these effects partially persist after post-training from a Base to a Chat model. This work advances mechanistic understanding of multilingual representations and offers a practical tool for probing neuron-level language organization with controlled interventions.

Abstract

Multilingual large language models (LLMs) achieve strong performance across languages, yet how language capabilities are organized at the neuron level remains poorly understood. Prior work has identified language-related neurons mainly through activation-based heuristics, which conflate language preference with functional importance. Prior work has identified language-related neurons mainly through activation-based heuristics, which conflate language preference with functional importance. We propose CRANE, a relevance-based analysis framework that redefines language specificity in terms of functional necessity, identifying language-specific neurons through targeted neuron-level interventions. CRANE characterizes neuron specialization by their contribution to language-conditioned predictions rather than activation magnitude. Our implementation will be made publicly available. Neuron-level interventions reveal a consistent asymmetric pattern: masking neurons relevant to a target language selectively degrades performance on that language while preserving performance on other languages to a substantial extent, indicating language-selective but non-exclusive neuron specializations. Experiments on English, Chinese, and Vietnamese across multiple benchmarks, together with a dedicated relevance-based metric and base-to-chat model transfer analysis, show that CRANE isolates language-specific components more precisely than activation-based methods.

Figures (6)

• Figure 1: Overview of CRANE. CRANE identifies language-specific neurons via relevance attribution and language-wise relevance statistics, and validates functional necessity through targeted neuron-level interventions with unified evaluation across NLU and open-ended benchmarks.
• Figure 2: Schematic intuition of kurtosis-based language contrast. Under a target language condition, a neuron may exhibit a more concentrated or heavy-tailed relevance distribution than under other languages. Kurtosis is used as one example statistic to quantify such concentration differences.
• Figure 3: Kurtosis-based contrast for identifying language-specific neurons. Each subplot corresponds to one Transformer layer. Each point represents a neuron, plotted by its normalized kurtosis under a target language (x-axis) versus the maximum kurtosis under other languages (y-axis). Dashed lines indicate the kurtosis threshold (set to 1). Neurons with high kurtosis for the target language but low kurtosis for others (bottom-right region) exhibit stronger language-specific relevance concentration.
• Figure 4: Targeted NLU degradation under neuron-level intervention on LLaMA2-7B-Base. The heatmap reports absolute performance drops ($\Delta = \text{Org} - \text{Masked}$) on each evaluation benchmark (columns) when masking neuron sets identified for a given target language and method (rows). Higher values indicate stronger degradation. The bar plot summarizes language-specific functional effects using LangSpec-F1.
• Figure 5: Transfer setting from Base to Chat. We first identify language-related neuron sets $\mathcal{N}_\ell$ on LLaMA2-7B-Base using relevance attribution and kurtosis-based statistics. The same neuron sets are then directly transferred and masked on LLaMA2-7B-Chatwithout re-identification to evaluate whether their functional influence persists after post-training.