Language Confusion Gate: Language-Aware Decoding Through Model Self-Distillation

TL;DR

The paper tackles language confusion in multilingual LLMs by introducing Language Confusion Gate (LCG), a lightweight, plug-in decoder intervention that masks disallowed language families without modifying the base model. It leverages norm-adjusted self-distillation and a token-language-family classifier to predict permissible languages at each step, significantly reducing CJ and Latin confusion across multiple models and benchmarks while preserving code-switching and task performance. The approach demonstrates strong empirical gains, shows compatibility with speculative decoding, and provides open-source datasets and code to enable reproducibility. This work offers a practical, scalable solution for reliable multilingual generation in both no-think and thinking modes, with clear directions for finer-grained future gates.

Abstract

Large language models (LLMs) often experience language confusion, which is the unintended mixing of languages during text generation. Current solutions to this problem either necessitate model retraining or cannot differentiate between harmful confusion and acceptable code-switching. This paper introduces the Language Confusion Gate (LCG), a lightweight, plug-in solution that filters tokens during decoding without altering the base LLM. The LCG is trained using norm-adjusted self-distillation to predict appropriate language families and apply masking only when needed. Our method is based on the findings that language confusion is infrequent, correct-language tokens are usually among the top predictions, and output token embedding norms are larger for high-resource languages, which biases sampling. When evaluated across various models, including Qwen3, GPT-OSS, Gemma3, Llama3.1, LCG decreases language confusion significantly, often by an order of magnitude, without negatively impacting task performance. Code is available at https://github.com/collinzrj/language_confusion_gate.

Figures (6)

• Figure 1: Left: examples of language confusion in three LLMs on the left. Right: LCG takes the LLM's hidden state, predicts permissible language families (CJ, Latin, Symbols, LowRes), and dynamically masks logits for disallowed tokens only when necessary to correct the confusion on the left without altering the base model.
• Figure 2: Top-10 Logits before and after applying norm at a confusion point on Qwen3-8B
• Figure 3: Comparison of LCG with baseline methods. CJ% and Latin% show CJ confusion rate and Latin confusion rate respectively.
• Figure 4: Prompt we used for ICL as described in Section \ref{['sec:icl_baseline']}
• Figure 5: Left: User asking in Arabic to write a python program that computes factorial. Right: User asking in English to let LLM introduce a Chinese poem.