Beyond English-Centric LLMs: What Language Do Multilingual Language Models Think in?
Chengzhi Zhong, Fei Cheng, Qianying Liu, Junfeng Jiang, Zhen Wan, Chenhui Chu, Yugo Murawaki, Sadao Kurohashi
TL;DR
This work probes how multilingual LLMs internally represent language via internal latent languages in their intermediate layers, using a logit-lens approach extended to multi-token Japanese processing. By analyzing Llama-2 (English-centric), Swallow (Japanese CPT), and LLM-jp (balanced bilingual), it shows that English-centric models rely on English as the latent pivot for Japanese tasks, while Japanese-focused models exhibit dual latent languages with probabilities aligned to the target language. The study also reveals culture-related latent biases in early layers that shift toward the target language in later layers, and that language-identity information resides in sparse dimensions separate from dense semantic features. These findings illuminate how training data shapes latent-language dynamics, with implications for improving multilingual robustness and guiding future interpretability and bias-mitigation efforts.
Abstract
In this study, we investigate whether non-English-centric LLMs, despite their strong performance, `think' in their respective dominant language: more precisely, `think' refers to how the representations of intermediate layers, when un-embedded into the vocabulary space, exhibit higher probabilities for certain dominant languages during generation. We term such languages as internal $\textbf{latent languages}$. We examine the latent language of three typical categories of models for Japanese processing: Llama2, an English-centric model; Swallow, an English-centric model with continued pre-training in Japanese; and LLM-jp, a model pre-trained on balanced English and Japanese corpora. Our empirical findings reveal that, unlike Llama2 which relies exclusively on English as the internal latent language, Japanese-specific Swallow and LLM-jp employ both Japanese and English, exhibiting dual internal latent languages. For any given target language, the model preferentially activates the latent language most closely related to it. In addition, we explore how intermediate layers respond to questions involving cultural conflicts between latent internal and target output languages. We further explore how the language identity shifts across layers while keeping consistent semantic meaning reflected in the intermediate layer representations. This study deepens the understanding of non-English-centric large language models, highlighting the intricate dynamics of language representation within their intermediate layers.