Less, but Better: Efficient Multilingual Expansion for LLMs via Layer-wise Mixture-of-Experts

TL;DR

LayerMoE tackles the efficiency and forgetting challenges in multilingual expansion of LLMs by introducing a layer-wise, similarity-driven allocation of new MoE experts and a routing classifier to protect old-language routing. By measuring language similarity across HSAs per layer, it assigns more experts to layers with lower similarity and fewer to more language-agnostic layers, while selectively adding a classifier to high-similarity layers to stabilize old-language routing. Empirically, LayerMoE achieves comparable or better performance than MoE-LPR/Aware baselines using up to 60% fewer new parameters in single-expansion and 33.3% fewer in lifelong-expansion, demonstrating a practical, resource-efficient path for continuous multilingual learning. The approach shows robust gains across multiple benchmarks and generalizes to other model families and translation tasks, underscoring its potential to scale multilingual capabilities with controlled forgetting.

Abstract

Continually expanding new languages for existing large language models (LLMs) is a promising yet challenging approach to building powerful multilingual LLMs. The biggest challenge is to make the model continuously learn new languages while preserving the proficient ability of old languages. To achieve this, recent work utilizes the Mixture-of-Experts (MoE) architecture to expand new languages by adding new experts and avoid catastrophic forgetting of old languages by routing corresponding tokens to the original model backbone (old experts). Although intuitive, this kind of method is parameter-costly when expanding new languages and still inevitably impacts the performance of old languages. To address these limitations, we analyze the language characteristics of different layers in LLMs and propose a layer-wise expert allocation algorithm (LayerMoE) to determine the appropriate number of new experts for each layer. Specifically, we find different layers in LLMs exhibit different representation similarities between languages and then utilize the similarity as the indicator to allocate experts for each layer, i.e., the higher similarity, the fewer experts. Additionally, to further mitigate the forgetting of old languages, we add a classifier in front of the router network on the layers with higher similarity to guide the routing of old language tokens. Experimental results show that our method outperforms the previous state-of-the-art baseline with 60% fewer experts in the single-expansion setting and with 33.3% fewer experts in the lifelong-expansion setting, demonstrating the effectiveness of our method.

Figures (5)

• Figure 1: The similarity of HSAs in different languages across all layers within Qwen1.5-1.8B.
• Figure 2: The procedure of our LayerMoE. In Step 1, we first calculate the indicated similarity for each layer. In Step 2, we allocate experts based on the similarity. The layers with lower similarity are allocated more experts, while those with higher similarity are assigned fewer experts. Additionally, we add a classifier in front of the router in the layers with higher similarity to guide the routing of old language tokens.
• Figure 3: The expert allocation of our method and baselines for each layer under G0 $\rightarrow$ G1, G0 $\rightarrow$ G2.
• Figure 4: The detailed similarity and expert allocation of our method for each layer of Qwen1.5-1.8B under the five settings.
• Figure 5: The detailed similarity and expert allocation of our method for each layer of Llama-3.2-3B under the G0 $\rightarrow$ G1 setting.