Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

Nandini Mundra, Aditya Nanda Kishore, Raj Dabre, Ratish Puduppully, Anoop Kunchukuttan, Mitesh M. Khapra

TL;DR

This paper establishes theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings.

Abstract

Language Models (LMs) excel in natural language processing tasks for English but show reduced performance in most other languages. This problem is commonly tackled by continually pre-training and fine-tuning these models for said languages. A significant issue in this process is the limited vocabulary coverage in the original model's tokenizer, leading to inadequate representation of new languages and necessitating an expansion of the tokenizer. The initialization of the embeddings corresponding to new vocabulary items presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods. We release our code publicly (https://github.com/AI4Bharat/VocabAdaptation_LLM/tree/CW2V).

An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models

TL;DR

This paper establishes theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings.

Abstract

Language Models (LMs) excel in natural language processing tasks for English but show reduced performance in most other languages. This problem is commonly tackled by continually pre-training and fine-tuning these models for said languages. A significant issue in this process is the limited vocabulary coverage in the original model's tokenizer, leading to inadequate representation of new languages and necessitating an expansion of the tokenizer. The initialization of the embeddings corresponding to new vocabulary items presents a further challenge. Current strategies require cross-lingual embeddings and lack a solid theoretical foundation as well as comparisons with strong baselines. In this paper, we first establish theoretically that initializing within the convex hull of existing embeddings is a good initialization, followed by a novel but simple approach, Constrained Word2Vec (CW2V), which does not require cross-lingual embeddings. Our study evaluates different initialization methods for expanding RoBERTa and LLaMA 2 across four languages and five tasks. The results show that CW2V performs equally well or even better than more advanced techniques. Additionally, simpler approaches like multivariate initialization perform on par with these advanced methods indicating that efficient large-scale multilingual continued pretraining can be achieved even with simpler initialization methods. We release our code publicly (https://github.com/AI4Bharat/VocabAdaptation_LLM/tree/CW2V).
Paper Structure (27 sections, 3 theorems, 11 equations, 12 figures, 4 tables)

This paper contains 27 sections, 3 theorems, 11 equations, 12 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Vocabulary Expansion
  5. What is a 'good' embedding initialization?
  6. Theorems
  7. Our Approach: Constrained Word2Vec
  8. Experimental Setting
  9. Models
  10. Tokenizers
  11. Datasets and Languages
  12. Baselines
  13. Constrained Word2Vec Training
  14. Downstream Tasks
  15. Results
  16. ...and 12 more sections

Key Result

Theorem 1

: A good initialization preserves the pre-expansion behavior. Let $e_1^s, e_2^s, e_3 ^s, ..., e_n^s \in \mathbb{R}^d$ be the embeddings of words in $\mathcal{V}^s$. Let $e_1^t,e_2^t, \hdots , e_{n'}^t \in \mathbb{R}^d$ be the embedding initializations for words in $\mathcal{V}^t$. If holds for all h $\in$$\mathbb{R}^d$, then $\left\{e^t_{j}; j \in \mathcal{V}^t \right\}$ is a 'good' initializatio

Figures (12)

  • Figure 1: Setup for Vocabulary Expansion. Source model is shown in blue blocks, and expanded vocabulary embeddings are represented in red blocks. Source model parameters remain unchanged.
  • Figure 2: Evaluation of different initialization methods on expanded RoBERTa models using three multilingual tasks (XNLI, NER, QA) at 15 CPT checkpoints. The plots show average performance across five languages.
  • Figure 3: 4-shot XNLI, MT, QA, XLSUM evaluation of different initialization methods on expanded LLaMA2 models at 5 equidistant CPT checkpoints. MT plots show average performance across 4 languages, and XNLI, QA, XLSUM plots show average performance across 5 languages.
  • Figure 4: Assessment of English performance for various initialization methods on expanded RoBERTa models across three downstream tasks (XNLI, NER, QA) at 15 CPT checkpoints.
  • Figure 5: Expanded LLaMA2 Model Outputs for the Prompt : "I don't want to eat" for various initializations.
  • ...and 7 more figures

Theorems & Definitions (6)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Theorem 3
  • proof