Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages

Wen Yang, Chong Li, Jiajun Zhang, Chengqing Zong

TL;DR

BigTranslate tackles the limited multilingual reach of open-source LLMs by augmenting LLaMA-13B with extensive Chinese-centric pretraining, a large-scale 102-language parallel corpus, and multilingual translation instruction tuning. The method combines a three-stage construction pipeline with an incremental curriculum to balance high-resource and low-resource languages, yielding a foundation model capable of translating across 102 languages. Evaluation with BLEU and GPT-4-based scoring shows BigTranslate is competitive with Google Translate and ChatGPT in many languages and outperforms ChatGPT in eight language pairs, demonstrating meaningful cross-lingual transfer and practical multilingual translation capabilities. The work highlights a viable path to extending LLMs for broad multilingual translation while acknowledging remaining gaps in extremely low-resource languages and suggesting future work to broaden applicability to other NLP tasks.

Abstract

Large language models (LLMs) demonstrate promising translation performance among various natural languages. However, many LLMs especially the open-sourced ones, such as BLOOM and LLaMA, are English-dominant and support only dozens of natural languages, making the potential of LLMs on language translation less explored. In this work, we present BigTranslate which adapts LLaMA that covers only 20 languages and enhances it with multilingual translation capability on more than 100 languages. BigTranslate is built upon LLaMA-13B and it is optimized in three steps. First, we continue training LLaMA with massive Chinese monolingual data. Second, we continue training the model with a large-scale parallel dataset that covers 102 natural languages. Third, we instruct-tune the foundation model with multilingual translation instructions, leading to our BigTranslate model. The preliminary experiments on multilingual translation show that BigTranslate performs comparably with ChatGPT and Google Translate in many languages and even outperforms ChatGPT in 8 language pairs. We release the BigTranslate model and hope it can advance the research progress.

BigTranslate: Augmenting Large Language Models with Multilingual Translation Capability over 100 Languages

TL;DR

BigTranslate tackles the limited multilingual reach of open-source LLMs by augmenting LLaMA-13B with extensive Chinese-centric pretraining, a large-scale 102-language parallel corpus, and multilingual translation instruction tuning. The method combines a three-stage construction pipeline with an incremental curriculum to balance high-resource and low-resource languages, yielding a foundation model capable of translating across 102 languages. Evaluation with BLEU and GPT-4-based scoring shows BigTranslate is competitive with Google Translate and ChatGPT in many languages and outperforms ChatGPT in eight language pairs, demonstrating meaningful cross-lingual transfer and practical multilingual translation capabilities. The work highlights a viable path to extending LLMs for broad multilingual translation while acknowledging remaining gaps in extremely low-resource languages and suggesting future work to broaden applicability to other NLP tasks.

Abstract

Large language models (LLMs) demonstrate promising translation performance among various natural languages. However, many LLMs especially the open-sourced ones, such as BLOOM and LLaMA, are English-dominant and support only dozens of natural languages, making the potential of LLMs on language translation less explored. In this work, we present BigTranslate which adapts LLaMA that covers only 20 languages and enhances it with multilingual translation capability on more than 100 languages. BigTranslate is built upon LLaMA-13B and it is optimized in three steps. First, we continue training LLaMA with massive Chinese monolingual data. Second, we continue training the model with a large-scale parallel dataset that covers 102 natural languages. Third, we instruct-tune the foundation model with multilingual translation instructions, leading to our BigTranslate model. The preliminary experiments on multilingual translation show that BigTranslate performs comparably with ChatGPT and Google Translate in many languages and even outperforms ChatGPT in 8 language pairs. We release the BigTranslate model and hope it can advance the research progress.
Paper Structure (22 sections, 4 figures, 3 tables, 1 algorithm)

This paper contains 22 sections, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Models
  4. Multilingual Neural Machine Translation
  5. BigTranslate Construction
  6. LLaMA as Foundation Model
  7. Augmenting Foundation Model with Chinese
  8. Augmenting Foundation Model with 102 languages
  9. Large-scale Parallel Dataset Construction
  10. Tokenizer
  11. Incremental Multilingual Pre-training
  12. Multilingual Translation Instruction Tuning
  13. Training Details
  14. Experiments
  15. Datasets and Evaluation Metrics
  16. ...and 7 more sections

Figures (4)

  • Figure 1: The language-pairs distribution of multilingual corpus. All the datasets consist of about 300 million sentence pairs.
  • Figure 2: The outline of three-step incremental multilingual pre-training approach. ① represents dividing multilingual language pairs into different intervals, ② denotes calculating sample means for all language pairs within each sample interval, ③ represents sorting the intervals in descending order based on sample mean values. The algorithm in step 3 is detailed in Algorithm \ref{['alg:Framwork']} for incremental pre-training.
  • Figure 3: An illustrated comparison of 102 languages from X to English or Chinese between BigTranslate, ChatGPT, and Google Translate. We sort the language scores in BLEU for BigTranslate in descending order.
  • Figure 4: An illustrated comparison of 70 languages from X to English or Chinese between BigTranslate, ChatGPT, and Google Translate. We sort the language scores in GPT-4 score for BigTranslate in descending order.