Language Lives in Sparse Dimensions: Toward Interpretable and Efficient Multilingual Control for Large Language Models

TL;DR

This work reveals that multilingual generation in English-centric LLMs is governed by a small, sparse set of language-specific dimensions that steer the transition from a shared, English-aligned intermediate representation to target-language token spaces. It introduces a training-free method to identify these dimensions using only ~50 sentences and applies a single-layer inference-time intervention to overwrite the identified dimensions with scaled language-specific signals, effectively switching output language while preserving semantic content. Across multiple models and five target languages, the approach outperforms neuron-based baselines in multilingual generation control, with the parallel data setting generally yielding stronger performance. The findings offer a scalable, interpretable mechanism for cross-lingual control and highlight the potential for efficient, data-efficient manipulation of LLM behavior without extensive fine-tuning or data requirements.

Abstract

Large language models exhibit strong multilingual capabilities despite limited exposure to non-English data. Prior studies show that English-centric large language models map multilingual content into English-aligned representations at intermediate layers and then project them back into target-language token spaces in the final layer. From this observation, we hypothesize that this cross-lingual transition is governed by a small and sparse set of dimensions, which occur at consistent indices across the intermediate to final layers. Building on this insight, we introduce a simple, training-free method to identify and manipulate these dimensions, requiring only as few as 50 sentences of either parallel or monolingual data. Experiments on a multilingual generation control task reveal the interpretability of these dimensions, demonstrating that the interventions in these dimensions can switch the output language while preserving semantic content, and that it surpasses the performance of prior neuron-based approaches at a substantially lower cost.

Figures (11)

• Figure 1: Hypothesis of language-specific dimensions. The logit lens shows that English-centric LLMs often surface the English answer in intermediate layers before producing the target-language answer. The prompt is "English:"Today is hot."-日本語:"今日は". We ask the model to predict the omitted word “hot” in Japanese. Dimensions in red are likely to be language-specific.
• Figure 2: Two settings for identifying language-specific dimensions. Using Japanese as an example, we first average token representations within each sentence to obtain a sentence-level mean vector. Then, compare the Japanese and English means and keep the top-$K$ dimensions with the largest absolute differences as Japanese-related dimensions. We demonstrate a real difference example of LLaMA2-13B on the left top.
• Figure 3: Selecting a top-$K$ threshold for detecting language-specific dimensions. Ablation study conducted on Llama2-7B in the parallel setting.
• Figure 4: Choosing an anchor layer for monolingual setting. Ablation study conducted on Llama2-7B in the monolingual setting.
• Figure 5: Overlap rate of language-specific dimensions identified by monolingual and parallel settings for different $K$. Experiments are conducted on Llama2-7B.