Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

Hao Zhou, Zhijun Wang, Shujian Huang, Xin Huang, Xue Han, Junlan Feng, Chao Deng, Weihua Luo, Jiajun Chen

TL;DR

MoE-LPR tackles the problem of expanding multilingual capabilities in LLMs without catastrophic forgetting of original languages. It introduces a two-stage approach: post-pretraining by upcycling the model into a Mixture-of-Experts (MoE) while freezing original parameters, and a lightweight review with Language Priors Routing (LPR) that uses minimal replay data to recover original-language performance. The method yields superior expansion performance while preserving original-language proficiency, maintains fixed inference overhead despite added experts, and demonstrates strong scalability and generalization to unseen languages. The work provides a practical, data-efficient path toward inclusive multilingual NLP systems with broad applicability.

Abstract

Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at https://github.com/zjwang21/MoE-LPR.git.

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

TL;DR

MoE-LPR tackles the problem of expanding multilingual capabilities in LLMs without catastrophic forgetting of original languages. It introduces a two-stage approach: post-pretraining by upcycling the model into a Mixture-of-Experts (MoE) while freezing original parameters, and a lightweight review with Language Priors Routing (LPR) that uses minimal replay data to recover original-language performance. The method yields superior expansion performance while preserving original-language proficiency, maintains fixed inference overhead despite added experts, and demonstrates strong scalability and generalization to unseen languages. The work provides a practical, data-efficient path toward inclusive multilingual NLP systems with broad applicability.

Abstract

Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at https://github.com/zjwang21/MoE-LPR.git.
Paper Structure (30 sections, 11 equations, 6 figures, 7 tables)

This paper contains 30 sections, 11 equations, 6 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Post-pretraining with MoE
  4. Review with LPR
  5. Experiments
  6. Experiment Setup
  7. Model and Languages
  8. Details of Post-pretraining
  9. Details of Review
  10. Baselines
  11. Evaluation Details
  12. Experiment Results
  13. Abalation & Analysis
  14. Review with LPR
  15. Performance Gain from Review & EC
  16. ...and 15 more sections

Figures (6)

  • Figure 1: MoE-LPR performs the best in both expanded languages and original languages. We define expanded languages as languages that the model is not very good at and we are going to enhance, and original languages as languages that the model is relatively strong in and prone to catastrophic forgetting.
  • Figure 2: Overall framework of our MoE-LPR. Two-stage strategy is performed to enhance the multilingual capability.
  • Figure 3: Router scores of the frozen expert for English (original language) tokens in the Belebele benchmark.
  • Figure 4: Average scores in expanded and original languages with varying numbers of documents for review.
  • Figure 5: Average scores in expanded and original languages with different model settings. "34.36" and "53.97" refer to the expanded and original language performance of the base model respectively.
  • ...and 1 more figures