Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models

TL;DR

The paper addresses the language gap between English-centric large language models and non-English languages by introducing EEVE, a vocabulary expansion method that combines parameter freezing and subword-based embedding initialization. The authors implement EEVE-Korean-v1.0, built on SOLAR-10.7B and Phi-2, with a seven-stage training process and a carefully initialized tokenizer to efficiently integrate a large Korean vocabulary while preserving English capabilities. Empirical results on KoBEST and English benchmarks show that EEVE-Korean-10.8B-v1.0 surpasses similar-sized open Korean models and maintains parity in English tasks, highlighting the method’s efficiency and effectiveness. The work contributes open-source Korean LLMs and demonstrates a scalable approach to multilingual adaptation that could extend to additional languages and evaluation tasks.

Abstract

This report introduces \texttt{EEVE-Korean-v1.0}, a Korean adaptation of large language models that exhibit remarkable capabilities across English and Korean text understanding. Building on recent highly capable but English-centric LLMs, such as SOLAR-10.7B and Phi-2, where non-English texts are inefficiently processed with English-centric tokenizers, we present an efficient and effective vocabulary expansion (EEVE) method, which encompasses parameter freezing and subword initialization. In contrast to previous efforts that believe new embeddings require trillions of training tokens, we show that our method can significantly boost non-English proficiency within just 2 billion tokens. Surpassing most instruction-tuned LLMs on the Open Ko-LLM Leaderboard, as of January 2024, our model \texttt{EEVE-Korean-10.8B-v1.0} ranks as the leading Korean pre-trained model in the open-source community, according to Hugging Face's leaderboard. We open-source our models on Huggingface to empower the open research community in various languages.

Figures (2)

• Figure 1: Training stages with parameter freezing. The fire and snowflake emojis indicate the trainable and frozen parameters respectively.
• Figure 2: An illustrative example of showing how our subword-based embedding initialization enables harmonize the old and new tokens at Stage 1. In Stage 1, the output embeddings of newly added tokens are initialized with the output embeddings of their first subword tokens that make up these new tokens, such that the last hidden representation for predicting "하세요" yields the same logits for the newly added token "하세요" with its first subword token "하". Even if we give the new token "하세요" as a gold token, the gradients are eventually computed based on its subword token "하", so the model takes the input embeddings of "하세요" to predict its subword "하".