Language Steering for Multilingual In-Context Learning

TL;DR

This work tackles the persistent cross-lingual gap in multilingual LLMs during in-context learning by introducing language vectors, a training-free steering mechanism that shifts model activations toward the target language during inference. By computing language-specific steering vectors from activation differences between parallel source and target language samples at intermediate layers, the method steers hidden representations via $\mathbf{h}_p'(t) = \mathbf{h}_p(t) + \alpha \cdot \mathbf{v}(t)$ without updating parameters. Empirical evaluation across MGSM, MSVAMP, and XNLI, over 19 languages and three model families, shows consistent improvements, particularly on structured mathematical reasoning, and reveals that these vectors encode linguistically meaningful structure aligned with language families. The analysis also highlights that steering transfers across tasks with varying success and that the approach is most effective in middle layers, offering a practical tool to narrow cross-lingual gaps while inviting considerations of safety and ethical deployment in multilingual AI applications.

Abstract

While multilingual large language models have gained widespread adoption, their performance on non-English languages remains substantially inferior to English. This disparity is particularly evident in in-context learning scenarios, where providing demonstrations in English but testing on non-English inputs leads to significant performance degradation. In this paper, we hypothesize that LLMs develop a universal semantic space for understanding languages, where different languages are encoded as distinct directions within this space. Based on this hypothesis, we propose language vectors -- a training-free language steering approach that leverages activation differences between source and target languages to guide model behavior. We steer the model generations by adding the vector to the intermediate model activations during inference. This is done to make the model's internal representations shift towards the target language space without any parameter updates. We evaluate our method across three datasets and test on a total of 19 languages on three different models. Our results show consistent improvements on multilingual in-context learning over baselines across all tasks and languages tested. Beyond performance gains, hierarchical clustering of steering vectors reveals meaningful linguistic structure aligned with language families. These vectors also successfully transfer across tasks, demonstrating that these representations are task-agnostic.

Figures (2)

• Figure 1: Overview of our language steering approach for multilingual in-context learning.Figure 1. Overview of our language steering approach for multilingual in-context learning. Step 1 (Extraction): We extract hidden states from parallel source and target language question-answer pairs at layer $t$ of the LLM. Step 2 (Vector Computation): The language steering vector $\mathbf{v}(t)$ is computed offline as mean of activation differences between target and source language representations across $N$ samples. Step 3 (Inference-Time Steering): During inference with few-shot demonstrations from source language and a target language question, we steer the model by adding $\alpha \cdot \mathbf{v}(t)$ to the hidden states at layer $t$, guiding the model toward target language reasoning patterns.
• Figure 2: Hierarchical clustering based on cosine distance between their steering vectors at layer 10 of Llama-3.1-8B-Instruct. Edge labels show cosine distances (lower = more similar). Colors indicate language families. This structure suggests that the model's internal representations encode meaningful language-specific patterns that align with both typological similarity and script characteristics.