Exploring Design Choices for Building Language-Specific LLMs

TL;DR

It is found that the initial performance of LLM does not always correlate with the final performance after the adaptation, and the simplest embedding initialization works well across various experimental settings.

Abstract

Despite rapid progress in large language models (LLMs), their performance on a vast majority of languages remains unsatisfactory. In this paper, we study building language-specific LLMs by adapting monolingual and multilingual LLMs. We conduct systematic experiments on how design choices (base model selection, vocabulary extension, and continued pretraining) impact the adapted LLM, both in terms of efficiency (how many tokens are needed to encode the same amount of information) and end task performance. We find that (1) the initial performance of LLM does not always correlate with the final performance after the adaptation. Adapting an English-centric models can yield better results than adapting multilingual models despite their worse initial performance on low-resource languages. (2) Efficiency can easily improved with simple vocabulary extension and continued pretraining in most LLMs we study, and (3) The optimal adaptation method (choice of the base model, new vocabulary size, training data, initialization strategy) is highly language-dependent, and the simplest embedding initialization works well across various experimental settings. Together, our work lays foundations on efficiently building language-specific LLMs by adapting existing LLMs.

Figures (6)

• Figure 1: Building a language-specific language model. After selecting a base language model (LM), we adapt it with two main steps: 1) Token Augmentation, which primarily involves extending the tokenizer to a target vocabulary size, and 2) Continued Pre-Training on the target corpus.
• Figure 2: Efficiency evaluation: the impact of vocabulary extension on the average sequence length. The shorter sequence length is more desirable. (a) extending the vocabulary with 10K tokens makes the token length substantially shorter, on par with that of English. (b) Adding more tokens continue to improve the sequence length with diminishing returns. (c) Performance (spBLEU) on FLORES machine translation versus maximum generation cutoff length; adapted models require 2$\times$ less tokens to achieve high performance.
• Figure 3: Change in performance ($\Delta$spBLEU) after adaptation. We report average performance across all benchmarks for two language pairs hi/ta, with continued training on 100K examples and vocabulary extension of $\Delta V$=10K tokens. Larger models are represented with bigger markers. Absolute numbers on all benchmarks are provided in \ref{['table:model_choice_hi']} and \ref{['table:model_choice_ta']} in \ref{['appdx:additional_exp']}.
• Figure 4: Adapted LLM's performance on Hindi/Tamil on generation and understanding benchmarks with increasing compute (measured in terms of hours per GPU). For models with extended vocabulary, $\Delta V$=50K.
• Figure 5: Performance variation on Hindi benchmarks with increasing CPT data (#examples). For models with extended vocabulary (LLaMA, Mistral), $\Delta V$=50K.