Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LLaMA Beyond English: An Empirical Study on Language Capability Transfer

Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, Xuanjing Huang

TL;DR

This study probes how to transfer LLaMA's language capabilities from English to non-English languages with minimal data and compute. It systematically evaluates vocabulary extension, further pretraining, and instruction tuning, using Chinese as the starting point and testing across 13 low-resource languages. Key findings show that vocabulary extension offers little benefit at tens-of-billions-scale, while effective transfer largely hinges on targeted pretraining and extensive instruction tuning, with multilingual training helping preserve English capabilities. The work provides practical guidance for building non-English LLMs at substantially reduced cost and demonstrates consistent trends across multiple language families.

Abstract

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (e.g. LLaMA) are pretrained on English-dominant corpus, which limits their performance in other non-English languages. In this paper, we focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language. To answer this question, we conduct an extensive empirical investigation based on LLaMA, accumulating over 1440 GPU hours. We analyze the impact of key factors such as vocabulary extension, further pretraining, and instruction tuning on transfer. To accurately assess the model's level of knowledge, we employ four widely used standardized testing benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a comprehensive evaluation of the model's response quality is conducted, considering aspects such as accuracy, fluency, informativeness, logical coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting instruction tasks from 17 diverse categories. Our evaluation results demonstrate that comparable performance to state-of-the-art transfer models can be achieved with less than 1% of the pretraining data, both in terms of knowledge alignment and response quality. Furthermore, the experimental outcomes across the thirteen low-resource languages also exhibit similar trends. We anticipate that the conclusions revealed by the experiments will aid the community in developing non-English LLMs.

LLaMA Beyond English: An Empirical Study on Language Capability Transfer

TL;DR

This study probes how to transfer LLaMA's language capabilities from English to non-English languages with minimal data and compute. It systematically evaluates vocabulary extension, further pretraining, and instruction tuning, using Chinese as the starting point and testing across 13 low-resource languages. Key findings show that vocabulary extension offers little benefit at tens-of-billions-scale, while effective transfer largely hinges on targeted pretraining and extensive instruction tuning, with multilingual training helping preserve English capabilities. The work provides practical guidance for building non-English LLMs at substantially reduced cost and demonstrates consistent trends across multiple language families.

Abstract

In recent times, substantial advancements have been witnessed in large language models (LLMs), exemplified by ChatGPT, showcasing remarkable proficiency across a range of complex tasks. However, many mainstream LLMs (e.g. LLaMA) are pretrained on English-dominant corpus, which limits their performance in other non-English languages. In this paper, we focus on how to effectively transfer the capabilities of language generation and following instructions to a non-English language. To answer this question, we conduct an extensive empirical investigation based on LLaMA, accumulating over 1440 GPU hours. We analyze the impact of key factors such as vocabulary extension, further pretraining, and instruction tuning on transfer. To accurately assess the model's level of knowledge, we employ four widely used standardized testing benchmarks: C-Eval, MMLU, AGI-Eval, and GAOKAO-Bench. Furthermore, a comprehensive evaluation of the model's response quality is conducted, considering aspects such as accuracy, fluency, informativeness, logical coherence, and harmlessness, based on LLM-Eval, a benchmarks consisting instruction tasks from 17 diverse categories. Our evaluation results demonstrate that comparable performance to state-of-the-art transfer models can be achieved with less than 1% of the pretraining data, both in terms of knowledge alignment and response quality. Furthermore, the experimental outcomes across the thirteen low-resource languages also exhibit similar trends. We anticipate that the conclusions revealed by the experiments will aid the community in developing non-English LLMs.
Paper Structure (20 sections, 2 equations, 4 figures, 3 tables)

This paper contains 20 sections, 2 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Overview
  3. Step 1: Pretraining to acquire language capability and knowledge
  4. Step 2: Instruction tuning for aligning with human intent
  5. Extrapolating LLMs to non-English languages
  6. Experimental Setup
  7. Models
  8. Datasets
  9. Evaluation Protocol
  10. Main Results
  11. The Impact of Vocabulary Extension on Transfer
  12. Training Scales Required for Effective Transfer
  13. Training Scales Required for Effective Transfer
  14. How about the Original English Capabilities
  15. Extending the Analysis to Multiple Languages
  16. ...and 5 more sections

Figures (4)

  • Figure 1: Pretrained LLaMA models, which are primarily trained on English-dominated corpus (as depicted on the left), are not inherently proficient in handling non-English languages. We aim to investigate the necessity of vocabulary extension, further pretraining, and instruction tuning, as well as to what extent they influence the capability transfer. This exploration enables us to efficiently transfer LLaMA's language capabilities to non-English languages (as illustrated on the right), minimizing costs in the process.
  • Figure 2: Knowledge-level evaluation results on four benchmarks.
  • Figure 3: Case study of code-switching. Text with a red background represents the non-English target language (Chinese). Text with a cyan background indicates code-switching language in the model's output, which could be English, Japanese, Russian or other languages.
  • Figure 4: Code-switching rate across languages.