Cross-Lingual Activation Steering for Multilingual Language Models

TL;DR

This work tackles persistent multilingual gaps in large language models by introducing Cross-Lingual Activation Steering (CLAS), a training-free, inference-time intervention that modulates neuron activations to rebalance shared and language-specific representations in a small set of bridge layers. CLAS operates through a three-stage pipeline—parallel input construction, neuron statistics-based categorization, and a lightweight steering rule—that blends modified activations with the original signal, leaving the anchor language unchanged. Across LLaMA 3.1 and Qwen 2.5, CLAS yields average improvements on XNLI and XQuAD, with gains driven by functional divergence rather than closer alignment to English, and is accompanied by rich analyses of representation reshaping and alignment. The findings suggest that targeted activation steering can unlock latent cross-lingual capacity without weight updates, offering a practical, data-free tool for improving non-dominant language performance in multilingual LLMs; future work could explore adaptive steering and extensions to other modalities.

Abstract

Large language models exhibit strong multilingual capabilities, yet significant performance gaps persist between dominant and non-dominant languages. Prior work attributes this gap to imbalances between shared and language-specific neurons in multilingual representations. We propose Cross-Lingual Activation Steering (CLAS), a training-free inference-time intervention that selectively modulates neuron activations. We evaluate CLAS on classification and generation benchmarks, achieving average improvements of 2.3% (Acc.) and 3.4% (F1) respectively, while maintaining high-resource language performance. We discover that effective transfer operates through functional divergence rather than strict alignment; performance gains correlate with increased language cluster separation. Our results demonstrate that targeted activation steering can unlock latent multilingual capacity in existing models without modification to model weights.

Figures (8)

• Figure 1: Distribution of types of neuron per layer across models. Llama has a total of 32 layers and Qwen has 28 layers. The bridge layers (purple shade) are selected near the final layers where the partial-shared language activations are the highest.
• Figure 2: Cosine similarity with English across languages on each task using Llama model. Similar results (Appendix \ref{['sec:appendix']}) were obtained with Qwen model.
• Figure 3: Relationship between alignment change to English (y-axis) and CLAS performance improvement (x-axis) for each language across different tasks and models.
• Figure 4: t-SNE visualization of cross-lingual representations before (left) and after (right) CLAS intervention. Each point represents a sentence embedding, with colors denoting languages. English, marked with stars, is the anchor language.
• Figure 5: Optimal value of $\alpha$ depends on the model and the task.