Language Drift in Multilingual Retrieval-Augmented Generation: Characterization and Decoding-Time Mitigation

TL;DR

The paper tackles language drift in multilingual Retrieval-Augmented Generation, where outputs often default to English or mix languages when the retrieved context is in a different language. It shows through controlled multilingual dataset construction and decoding analyses that drift stems from decoder-level biases rather than comprehension failure, and that English acts as a dominant semantic attractor across languages. As a remedy, the authors introduce Soft-Constrained Decoding (SCD), a training-free, decoding-time approach that softly penalizes non-target-language tokens using a three-way token categorization and logits adjustment with parameters $\alpha>1$, $\beta<1$, and a cold-start threshold $T_{\text{start}}$, making SCD model- and task-agnostic. Empirical results across HotpotQA, MuSiQue, and DuReader with multiple backbones show that SCD improves both language consistency (LC) and generation quality (BLEU/ROUGE) compared to prompting or hard-vocabulary constraints, promoting longer and more coherent reasoning while maintaining target-language outputs. The work highlights the practical value of decoding-time language control for multilingual RAG and provides a generalizable framework for mitigating language drift in real-world applications.

Abstract

Multilingual Retrieval-Augmented Generation (RAG) enables large language models (LLMs) to perform knowledge-intensive tasks in multilingual settings by leveraging retrieved documents as external evidence. However, when the retrieved evidence differs in language from the user query and in-context exemplars, the model often exhibits language drift by generating responses in an unintended language. This phenomenon is especially pronounced during reasoning-intensive decoding, such as Chain-of-Thought (CoT) generation, where intermediate steps introduce further language instability. In this paper, we systematically study output language drift in multilingual RAG across multiple datasets, languages, and LLM backbones. Our controlled experiments reveal that the drift results not from comprehension failure but from decoder-level collapse, where dominant token distributions and high-frequency English patterns dominate the intended generation language. We further observe that English serves as a semantic attractor under cross-lingual conditions, emerging as both the strongest interference source and the most frequent fallback language. To mitigate this, we propose Soft Constrained Decoding (SCD), a lightweight, training-free decoding strategy that gently steers generation toward the target language by penalizing non-target-language tokens. SCD is model-agnostic and can be applied to any generation algorithm without modifying the architecture or requiring additional data. Experiments across three multilingual datasets and multiple typologically diverse languages show that SCD consistently improves language alignment and task performance, providing an effective and generalizable solution in multilingual RAG.

Figures (10)

• Figure 1: Illustration of language drift in multilingual RAG. The user query and in-context examples are provided in the target language (e.g., Chinese), while the retrieved context is written in a non-target language (e.g., English). During reasoning, the model mixes languages and ultimately outputs the final answer in a non-target language.
• Figure 2: Relative performance gap between with-ICL and without-ICL settings across different target and context language combinations on the HotpotQA dataset, visualized as a polar radar chart. Each axis corresponds to one evaluation metric (ROUGE, BLEU, or LC) under a specific context language (EN, ZH, AR, or RU), totaling 12 axes. Solid lines represent different target languages, color-coded accordingly. Values indicate the percentage difference between ICL and non-ICL performance under each configuration. The black dashed ring at 0% denotes no change. Labels mark the highest gains and the most severe LC degradations. The chart reveals that ICL consistently improves BLEU and ROUGE, but often reduces language consistency, especially under ZH and RU contexts.
• Figure 3: Language drift patterns on the HotpotQA dataset for LLaMA3-8B and Qwen2.5-7B models. Each cell shows the percentage of inconsistent outputs that are generated in English (EN). Both models exhibit a strong fallback tendency toward English across all cross-lingual settings.
• Figure 4: Distribution of four language drift types across different target–context language pairs in the HotpotQA dataset. Each subplot corresponds to a fixed target language (ZH, AR, RU), with the x-axis denoting the context language. The top row displays results for LLaMA3-8B, and the bottom row for Qwen2.5-7B.
• Figure 5: Language drift patterns on the Musique dataset for LLaMA3-8B and Qwen2.5-7B models. Each cell shows the percentage of inconsistent outputs that are generated in English.