Model-Aware Tokenizer Transfer

TL;DR

This paper tackles the tokenizer bottleneck in multilingual LLM adaptation by introducing Model-Aware Tokenizer Transfer (MATT), which leverages an Attention Influence Modeling (AIM) objective to distill inter-token attention patterns from a source model to a target model with a new tokenizer. By aligning attention dynamics rather than relying solely on embedding similarity, MATT provides an efficient warm-up that substantially recovers the original model’s performance across generative and discriminative tasks, often with only a few GPU hours. The approach is validated on multiple languages and model families, notably achieving strong results for Ukrainian with Gemma 3, and demonstrating favorable efficiency-complexity trade-offs via layer-wise AIM application and optional AIM* variants. These findings suggest that incorporating model-level signals into tokenizer transfer offers a practical path for robust multilingual adaptation without full re-training.

Abstract

Large Language Models (LLMs) are trained to support an increasing number of languages, yet their predefined tokenizers remain a bottleneck for adapting models to lower-resource or distinct-script languages. Existing tokenizer transfer methods typically rely on semantic heuristics to initialize new embeddings, ignoring higher-layer model dynamics and limiting transfer quality. We propose Model-Aware Tokenizer Transfer (MATT), a method that incorporates model internals into the tokenizer transfer process. MATT introduces an Attention Influence Modeling (AIM) objective that distills inter-token communication patterns from a source model into a target model with a new tokenizer, providing an efficient warm-up before standard language modeling. Unlike approaches that focus solely on embedding similarity, MATT leverages attention behavior to guide embedding initialization and adaptation. Experiments across diverse linguistic settings show that MATT recovers a large fraction of the original model's performance within a few GPU hours, outperforming heuristic baselines. These results demonstrate that incorporating model-level signals offers a practical and effective path toward robust tokenizer transfer in multilingual LLMs.

Figures (5)

• Figure 1: Attention Influence Modeling (AIM) objective with word segmentation. For each input, the weighted value vectors ${\bm{v}}^*_{i,j}$ of the original tokens $t_j$ are aggregated into segment-level vectors $\bm{\mathfrak{s}}_{i, k}$ according to a chosen word-segmentation function. The model trained with the new tokenizer produces its own segment representations $\bm{\mathfrak{s}}^{\prime}_{i, k}$. The AIM objective encourages these new segment representations to stay close to the segment representations $\bm{\mathfrak{s}}_{i, k}$ computed from the model using the old tokenizer. All this happens with respect to the query state $\bm{\mathfrak{q}}_5$ of the 5th segment (_methane), which is equal to the query state of its last token -- hane.
• Figure 2: Token-level attention alignment between teacher and student models. The left matrix shows the weighted value states of the teacher model using the original tokenizer $T$, and the right matrix shows those of the student model using the new tokenizer $T^{\prime}$. Each square represents the weighted value state ${\bm{v}}^*_{i,j}$ of $t_j$ for query token $t_i$ ($i$ for rows, $j$ for columns). Numbers (or matching colors) within a matrix identify tokens that are aggregated into the same segment-level state $\bm{\mathfrak{s}}_{i, k}$. Numbers (or matching colors) across the two matrices indicate corresponding pairs $\bm{\mathfrak{s}}_{\ell_T(i), j}$ and $\bm{\mathfrak{s}}^{\prime}_{\ell_{T^{\prime}}(i), j}$ used in the loss $\mathcal{L}^*$ to align the teacher and the student attention representations.
• Figure 3: Offset-based segmentation algorithm visualization.
• Figure 4: Accuracy on the Belebele benchmark over training tokens for five languages, with horizontal lines marking FOCUS initialization and full performance.
• Figure 5: Effect of applying the AIM objective to different numbers of layers. The plots show the trade-off between model performance (a) and computational efficiency (b) as the application depth increases.