Adapters for Altering LLM Vocabularies: What Languages Benefit the Most?

TL;DR

The paper introduces VocADT, a vocabulary adaptation method that uses adapters to learn the optimal linear combination of existing embeddings for new vocabulary while keeping the base LLM fixed, enabling scalable multilingual expansion without external resources. Across 11 languages with diverse scripts, VocADT consistently improves over the original model and strong baselines, with Latin-script and highly fragmented languages benefiting the most. The approach remains effective after downstream fine-tuning for machine translation, and generalizes to other models (e.g., LLaMA) and larger language groups. Together, these results demonstrate a flexible, resource-efficient path to extending LLM vocabularies for multilingual and cross-lingual generation tasks.

Abstract

Vocabulary adaptation, which integrates new vocabulary into pre-trained language models, enables expansion to new languages and mitigates token over-fragmentation. However, existing approaches are limited by their reliance on heuristics or external embeddings. We propose VocADT, a novel method for vocabulary adaptation using adapter modules that are trained to learn the optimal linear combination of existing embeddings while keeping the model's weights fixed. VocADT offers a flexible and scalable solution without depending on external resources or language constraints. Across 11 languages-with diverse scripts, resource availability, and fragmentation-we demonstrate that VocADT outperforms the original Mistral model and other baselines across various multilingual tasks including natural language understanding and machine translation. We find that Latin-script languages and highly fragmented languages benefit the most from vocabulary adaptation. We further fine-tune the adapted model on the generative task of machine translation and find that vocabulary adaptation is still beneficial after fine-tuning and that VocADT is the most effective.

Figures (6)

• Figure 1: Overview of our vocabulary adaptation with adapter (VocADT) and the initialization of adapter. The vocabulary adapter modules are trained to adapt new vocabulary with existing embeddings while keeping the original model fixed. We initialize entries of the adapter for overlapping tokens and tokens whose partitions are in the original vocabulary. Once trained, the adapters and original embeddings are merged to form the new embeddings.
• Figure 2: Average scores of original Mistral and its adaptation with new vocabulary, only replacing embeddings and fixing the body of transformer modules. "-multi" indicates models with a multilingual vocabulary, which includes five languages covering all languages with two separate models, while "-mono" refers to monolingual vocabulary models. xx-en and en-xx indicate MT tasks. See Appendix \ref{['sec:apdx_result_number_langwise']} for individual values.
• Figure 3: Effect of vocabulary adaption on mitigating over-fragmentation and task performance. The $y$-axis for the increase rate on the left side is limited to the positive range. Languages with Latin scripts or those experiencing severe fragmentation benefit the most. xx-en and en-xx are machine translation tasks. See Appendix \ref{['sec:apdx_result_number_langwise']} for individual task performance values.
• Figure 4: Comparison of task performance between two grouping strategies of Mixed-script and Cyrillic-script on two shared languages. Consistent script within a group provides minor benefits.
• Figure 5: Effects of the auxiliary loss on various settings.