Lens: Rethinking Multilingual Enhancement for Large Language Models

TL;DR

Lens addresses the English-centric bias of state-of-the-art LLMs by leveraging internal language representation spaces rather than external multilingual data. It decomposes the multilingual latent space into a language-agnostic subspace $\boldsymbol{M}_a$ and a language-specific subspace $\boldsymbol{M}_s$ via Language Subspace Probing, then applies Language Subspace Manipulation with targeted losses to align target languages to English in $\boldsymbol{M}_a$ while pushing country-specific features in $\boldsymbol{M}_s$ along language-expression directions. The method updates only the upper layers, uses a central-language pivot to preserve proficiency, and demonstrates improved multilingual understanding and generation across bilingual and multilingual setups on three English-centric LLMs with lower computational overhead than post-training baselines. Lens outperforms baselines and open-source multilingual models on diverse benchmarks while maintaining central-language performance, highlighting a scalable, interpretable path to broader multilingual AI access. The work suggests a paradigm shift toward internal, representation-based supervision and points to future hybrids with data-driven signals for even stronger multilingual capabilities.

Abstract

As global demand for multilingual large language models (LLMs) grows, most LLMs still remain overly focused on English, leading to the limited access to advanced AI for non-English speakers. Current methods to enhance multilingual capabilities largely rely on data-driven post-training techniques, such as multilingual instruction tuning or continual pre-training. However, these approaches exhibit significant limitations, including high resource cost, exacerbation of off-target issue and catastrophic forgetting of central language abilities. To this end, we propose Lens, a novel approach that enhances multilingual capabilities by leveraging LLMs' internal language representation spaces. Lens operates on two subspaces: the language-agnostic subspace, where it aligns target languages with the central language to inherit strong semantic representations, and the language-specific subspace, where it separates target and central languages to preserve linguistic specificity. Experiments on three English-centric LLMs show that Lens significantly improves multilingual performance while maintaining the model's English proficiency, achieving better results with less computational cost compared to existing post-training approaches.

Figures (7)

• Figure 1: The overall architecture of our proposed Lens for multilingual enhancement. (1) In the LSP, we begin by decomposing the multilingual latent space, which is formed by the representations of probing samples from both the target and central languages. Using singular value decomposition (SVD), we separate this space into two orthogonal components: a language-agnostic subspace, $\boldsymbol{M}_{a}$, and a language-specific subspace, $\boldsymbol{M}_{s}$. (2) Then in LSM, the parallel multilingual representations of the target languages are pushed toward their respective linguistic expression directions within $\boldsymbol{M}_{s}$, while being pulled closer to the central language in $\boldsymbol{M}_{a}$. Additionally, the representations of the central language are carefully constrained to remain largely intact.
• Figure 2: The ablation results to verify the effectiveness and impact of different optimization objectives in LSM. MU Performance stands for the average performance on all multilingual understanding benchmarks, while MG Performance is the results on MT-Bench. LF represents language fidelity.
• Figure 3: (a) The impact of varying the number of manipulated layers. (b) The impact of training data volume. (c) Comparison with open-source multilingual-enhanced LLMs.
• Figure 4: The PCA visualization of multilingual representations projected in the obtained language-agnostic subspace (right) and the language-specific (left) subspace. The backbone model is LLaMA-3-8B-Instruct after multilingual enhanced with Lens.
• Figure 5: Results on the multilingual understanding and generation benchmarks with LLaMA-3-8B-Instruct backbone under the multilingual setting.