The Rise and Down of Babel Tower: Investigating the Evolution Process of Multilingual Code Large Language Model

TL;DR

This work investigates how multilingual capabilities in large language models emerge and evolve during the pre-training phase, using code-focused LLMs as a testbed. It introduces the Babel Tower Hypothesis, which posits that languages initially share a single dominant knowledge system and gradually form language-specific systems through three stages: Translation, Transition, and Stabilization. The authors validate the hypothesis by probing internal states with working-language analyses and language transferring neurons, observing stage-like shifts and cross-language transfer quantified by the proportion $|\u211d{P}ap{C}|/|{C}|$. They further show that optimizing pre-training data distribution yields substantial cross-lingual gains, offering practical guidance for multilingual pre-training and potential extensions to natural languages.

Abstract

Large language models (LLMs) have shown significant multilingual capabilities. However, the mechanisms underlying the development of these capabilities during pre-training are not well understood. In this paper, we use code LLMs as an experimental platform to explore the evolution of multilingual capabilities in LLMs during the pre-training process. Based on our observations, we propose the Babel Tower Hypothesis, which describes the entire process of LLMs acquiring new language capabilities. During the learning process, multiple languages initially share a single knowledge system dominated by the primary language and gradually develop language-specific knowledge systems. We then validate the above hypothesis by tracking the internal states of the LLMs through identifying working languages and language transferring neurons. Experimental results show that the internal state changes of the LLM are consistent with our Babel Tower Hypothesis. Building on these insights, we propose a novel method to construct an optimized pre-training corpus for multilingual code LLMs, which significantly outperforms LLMs trained on the original corpus. The proposed Babel Tower Hypothesis provides new insights into designing pre-training data distributions to achieve optimal multilingual capabilities in LLMs.

Figures (9)

• Figure 1: The evolution of a LLM learning a new language. In this figure, Python serves as the initial dominant language, with PHP as the new language. The process consists of three distinct stages: (1) Translation Stage: the performance of PHP improves rapidly and the Python system is dominated in the LLM, and PHP generation primarily relies on the Python system; (2) Transition Stage: the performance of PHP begins to decline while it gradually forms its own system; (3) Stabilization Stage: the performance of PHP stabilizes, and the generation of PHP depends on its own system.
• Figure 2: The performance of PHP/C# (left y-axis) and the proportion of correct answers requiring knowledge from the Python corpus (Knowledge transferred from Python, right y-axis) across training steps (x-axis). Based on the performance curve, we divide the entire process into three stages. The performance improves initially, then gradually declines, and eventually stabilizes. Concurrently, in the early stages, the proportion of correct answers requiring knowledge from the Python corpus is relatively high but subsequently decreases.
• Figure 3: The proportion of a language being the working language (WL proportion, y-axis) when generating new languages across training steps (x-axis). Throughout the pre-training process, the working languages gradually transition from Python to the new languages.
• Figure 4: The number of language transferring neurons in different languages (#LT neurons, y-axis) across training steps (x-axis). Throughout the pre-training process, the language transferring neurons for new languages activated gradually diminish, whereas those for Python activated progressively increase.
• Figure 5: The final internal states under different pre-training corpus. The X-axis represents the number of PHP tokens in the pre-training corpus, while the Y-axis indicates the performance, the proportion of the working languages, and the number of language transferring neurons corresponding to the final state.