Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral

Yiming Cui, Xin Yao

TL;DR

The paper tackles adapting a sparse Mixture-of-Experts LLM (Mixtral) to Chinese while preserving English performance. It introduces Chinese-Mixtral and Chinese-Mixtral-Instruct via continual Chinese pre-training and instruction fine-tuning using QLoRA, without extending the vocabulary. Through extensive benchmarks (C-Eval, CMMLU, Open LLM Leaderboard, LongBench) and a human chatbot arena, it demonstrates that instruction-tuned variants substantially improve Chinese abilities and can match or surpass English baselines on several tasks, while providing insights into vocabulary extension, starting-point initialization, and long-context generalization. The work offers practical guidance for language-transfer strategies in SMoE LLMs and contributes open-source resources for reproducibility and further research.

Abstract

Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance. Based on Mixtral-8x7B-v0.1, in this paper, we propose Chinese-Mixtral and Chinese-Mixtral-Instruct with improved Chinese language abilities by adopting further pre-training and instruction fine-tuning. Experimental results show that our Chinese-Mixtral and Chinese-Mixtral-Instruct successfully improve Chinese understanding and generation performance while retaining the original English abilities. Then, we discuss several key questions when performing language adaptation on large language models, including the necessity of extending the language-specific vocabulary and the choice of the initialization model (foundation model v.s. instruction model), by providing empirical results and analysis. We also present the visualizations of each expert to examine their importance on downstream tasks. Our resources are publicly available through \url{https://github.com/ymcui/Chinese-Mixtral}.

Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral

TL;DR

The paper tackles adapting a sparse Mixture-of-Experts LLM (Mixtral) to Chinese while preserving English performance. It introduces Chinese-Mixtral and Chinese-Mixtral-Instruct via continual Chinese pre-training and instruction fine-tuning using QLoRA, without extending the vocabulary. Through extensive benchmarks (C-Eval, CMMLU, Open LLM Leaderboard, LongBench) and a human chatbot arena, it demonstrates that instruction-tuned variants substantially improve Chinese abilities and can match or surpass English baselines on several tasks, while providing insights into vocabulary extension, starting-point initialization, and long-context generalization. The work offers practical guidance for language-transfer strategies in SMoE LLMs and contributes open-source resources for reproducibility and further research.

Abstract

Mixtral, a representative sparse mixture of experts (SMoE) language model, has received significant attention due to its unique model design and superior performance. Based on Mixtral-8x7B-v0.1, in this paper, we propose Chinese-Mixtral and Chinese-Mixtral-Instruct with improved Chinese language abilities by adopting further pre-training and instruction fine-tuning. Experimental results show that our Chinese-Mixtral and Chinese-Mixtral-Instruct successfully improve Chinese understanding and generation performance while retaining the original English abilities. Then, we discuss several key questions when performing language adaptation on large language models, including the necessity of extending the language-specific vocabulary and the choice of the initialization model (foundation model v.s. instruction model), by providing empirical results and analysis. We also present the visualizations of each expert to examine their importance on downstream tasks. Our resources are publicly available through \url{https://github.com/ymcui/Chinese-Mixtral}.
Paper Structure (15 sections, 4 equations, 5 figures, 9 tables)

This paper contains 15 sections, 4 equations, 5 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Chinese Mixtral
  3. Experimental Setups
  4. Experimental Results
  5. C-Eval
  6. CMMLU
  7. Open LLM Leaderboard
  8. LongBench
  9. Chinese LLM Chatbot Arena
  10. Discussion
  11. Effect of Extending Chinese Vocabulary
  12. Effect of the Initialization Model
  13. Effect of Long Context Abilities
  14. Visualizations
  15. Conclusion

Figures (5)

  • Figure 1: Architecture of Mixtral.
  • Figure 2: Interface of Chinese LLM chatbot arena.
  • Figure 3: Results of pair-wise winning rate and battle count in Chinese LLM chatbot arena.
  • Figure 4: Perplexities under different context lengths (on validation set).
  • Figure 5: Visualizations of each expert in each layer.