Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models

TL;DR

This paper tackles tokenizer adaptation for pretrained models by proposing continued BPE training for vocabulary extension and a leaf-based pruning strategy to shrink vocabularies without harming performance. Continued BPE training yields more efficient tokenization and better utilization of added tokens than naive extension, while leaf-based pruning preserves model quality during vocabulary reduction. Across multilingual experiments and several model families, the methods deliver substantial compression gains and enable safe pruning, with downstream results often unaffected or even improved. An open-source toolkit is released to enable practitioners to apply these techniques to domain- and language-specific tokenization needs.

Abstract

Tokenizer adaptation plays an important role in transferring pre-trained language models to new domains or languages. In this work, we address two complementary aspects of this process: vocabulary extension and pruning. The common approach to extension trains a new tokenizer on domain-specific text and appends the tokens that do not overlap with the existing vocabulary, which often results in many tokens that are unreachable or never used. We propose continued BPE training, which adapts a pre-trained tokenizer by continuing the BPE merge learning process on new data. Experiments across multiple languages and model families show that this approach improves tokenization efficiency and leads to better utilization of added vocabulary. We also introduce leaf-based vocabulary pruning, which removes redundant tokens while preserving model quality. Together, these methods provide practical tools for controlled vocabulary modification, which we release as an open-source package.

Figures (13)

• Figure 1: Change in text compression (bytes per token$\uparrow$) of Estonian text using the Llama 3 tokenizer extended with Estonian tokens. We compare naive tokenizer extension and continued BPE training (ours). The monolingual tokenizer is trained solely on Estonian data.
• Figure 2: Change in text compression (bytes per token$\uparrow$) for vocabulary pruning methods on Estonian-English pruning (Llama-3).
• Figure 3: The comparison of vocabulary extension methods: previously common approach of training a tokenizer and using its vocabulary for extending an existing tokenizer (left) and continuing BPE training on an existing tokenizer proposed by us (right).
• Figure 4: Llama-3.1-8B pruned with different methods and evaluated on Estonian and English downstream tasks. The dataset for training the pruner consisted of 50-50 Estonian-English mix. The highlighted area shows the 95% confidence interval acquired through bootstrap resampling.
• Figure 5: Extension training budget in characters when adding 16,000 tokens to the Llama-3 tokenizer.