Language Surgery in Multilingual Large Language Models

TL;DR

This work investigates naturally emerging representation alignment in multilingual LLMs, showing that middle-layer representations retain language-specific cues while maintaining cross-lingual alignment. It introduces Inference-Time Language Control (ITLC), a latent-intervention technique that extracts language vectors via Linear Discriminant Analysis from middle-layer states and injects a shift vector during generation to steer decoding toward a target language with minimal semantic loss. ITLC yields strong cross-lingual control on language confusion benchmarks and competitive semantic retention, often matching or approaching performance of more expensive test-time interventions while requiring only a single middle-layer intervention. The findings deepen understanding of how representation alignment relates to language-specific information and offer a practical, efficient tool to enhance multilingual capabilities of LLMs for robust cross-lingual generation.

Abstract

Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages, revolutionizing natural language processing. This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers, and its implications for disentangling language-specific and language-agnostic information. We empirically confirm the existence of this alignment, analyze its behavior in comparison to explicitly designed alignment models, and demonstrate its potential for language-specific manipulation without semantic degradation. Building on these findings, we propose Inference-Time Language Control (ITLC), a novel method that leverages latent injection to enable precise cross-lingual language control and mitigate language confusion in LLMs. Our experiments highlight ITLC's strong cross-lingual control capabilities while preserving semantic integrity in target languages. Furthermore, we demonstrate its effectiveness in alleviating the cross-lingual language confusion problem, which persists even in current large-scale LLMs, leading to inconsistent language generation. This work advances our understanding of representation alignment in LLMs and introduces a practical solution for enhancing their monolingual and cross-lingual performance.

Figures (13)

• Figure 1: We inspect the alignment in the middle layer representation of LLMs, allowing us to disentangle the language-specific and language-agnostic information. By exploiting this behavior, we are able to achieve Inference-Time Language Control (ITLC), alleviating the language confusion problem in LLMs.
• Figure 2: Cross-lingual similarity across different layers in LaBSE and Qwen2.5-0.5B. LaBSE exhibits high cross-lingual similarity in its final layer, whereas Qwen2.5-0.5B shows this similarity in the middle layer. This difference suggests that the alignment of representations occurs at distinct positions within the two models.
• Figure 3: Comparison of LPR metrics on LCB between Qwen2.5-7B-Instruct with ITLC and Qwen2.5-7b-EAX across 14 languages in monolingual and cross-lingual settings.
• Figure 4: Cross-lingual LPR performance on LCB, comparing base and instruct shift vector applications.
• Figure 5: Generation performance for different target languages on Qwen2.5 and Llama-3.1 Instruct models based on chrF++ (Left) and BERT F1 (Right). Baseline denotes the same model prompted in the same language as the desired target language.