Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models

TL;DR

The paper investigates how multilingual large language models internally support cross-lingual competence by mapping inputs into a Lingua Franca latent space and identifying language-specific linguistic regions. It introduces activation-similarity based metrics and a probing method to locate key linguistic regions, demonstrating that semantic alignment strengthens with training and model scale, and that key linguistic neurons concentrate in the first and last layers. Empirical results on BLOOM and LLaMA2 show that deactivating language-specific regions mainly harms the corresponding language, while semantically equivalent inputs from different languages converge in a common semantic space $S_{ij}$, with this convergence becoming more pronounced as models grow. These findings offer mechanistic insight into cross-lingual transfer and provide actionable guidance for improving multilingual LLMs' cross-language reasoning capabilities, including how to leverage early-layer encoding and semantic-space alignment in scaling and fine-tuning.

Abstract

Large language models (LLMs) have demonstrated remarkable performance, particularly in multilingual contexts. While recent studies suggest that LLMs can transfer skills learned in one language to others, the internal mechanisms behind this ability remain unclear. We observed that the neuron activation patterns of LLMs exhibit similarities when processing the same language, revealing the existence and location of key linguistic regions. Additionally, we found that neuron activation patterns are similar when processing sentences with the same semantic meaning in different languages. This indicates that LLMs map semantically identical inputs from different languages into a "Lingua Franca", a common semantic latent space that allows for consistent processing across languages. This semantic alignment becomes more pronounced with training and increased model size, resulting in a more language-agnostic activation pattern. Moreover, we found that key linguistic neurons are concentrated in the first and last layers of LLMs, becoming denser in the first layers as training progresses. Experiments on BLOOM and LLaMA2 support these findings, highlighting the structural evolution of multilingual LLMs during training and scaling up. This paper provides insights into the internal workings of LLMs, offering a foundation for future improvements in their cross-lingual capabilities.

Figures (11)

• Figure 1: LLMs encode inputs into a "Lingua Franca", a latent semantic space representation shared by all languages, and then decode this "Lingua Franca" into the target language. As training progresses and the models scale up, LLMs become better at mapping inputs to this common semantic space.
• Figure 2: Similarity map of the BLOOM-7b1 model. Each block of 100 samples is in the same language. Samples in the same language form distinct light blocks, and samples with the same semantic meaning form light bands along the diagonal of these blocks.
• Figure 3: Layer-wise Semantic Alignment Development Scores (SADS) and Linguistic Regions Development Scores (LRDS) of BLOOM-7B1.
• Figure 4: Comparison of the evolution of linguistic regions (left) and semantic alignment (right) with training steps. As training progresses, the key linguistic regions become smaller, and the neuron activation pattern becomes more language-agnostic. Meanwhile, semantic alignment becomes more pronounced, and the model's cross-lingual reasoning ability improves.
• Figure 5: Distribution of Key Neurons Across Different Layers. We may see that as the training steps grows, the key regions become denser in the first layers, facilitating the encoding from inputs to the "Lingua Franca".