TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

TL;DR

TokSuite directly isolates the impact of tokenizer design on language model behavior by training 14 identically initialized LMs that differ only in their tokenizers and by constructing a multilingual robustness benchmark spanning five languages and multiple domains. The authors additionally align vocabularies with a unified super vocabulary to enable fair cross-model initialization and evaluation, and they propose intrinsic efficiency metrics to quantify tokenization compactness and cross-lingual parity. Key findings show that tokenizer design can influence robustness as much as, or more than, vocabulary size, with byte-level tokenizers like ByT5 exhibiting strong multilingual resilience at the cost of efficiency, while certain perturbations drastically degrade performance in math and STEM content for many tokenizers. This work provides a principled framework and open resources for evaluating tokenizer-related effects, highlighting practical implications for tokenizer selection and future research into robust, language-agnostic tokenization strategies.

Abstract

Tokenizers provide the fundamental basis through which text is represented and processed by language models (LMs). Despite the importance of tokenization, its role in LM performance and behavior is poorly understood due to the challenge of measuring the impact of tokenization in isolation. To address this need, we present TokSuite, a collection of models and a benchmark that supports research into tokenization's influence on LMs. Specifically, we train fourteen models that use different tokenizers but are otherwise identical using the same architecture, dataset, training budget, and initialization. Additionally, we curate and release a new benchmark that specifically measures model performance subject to real-world perturbations that are likely to influence tokenization. Together, TokSuite allows robust decoupling of the influence of a model's tokenizer, supporting a series of novel findings that elucidate the respective benefits and shortcomings of a wide range of popular tokenizers.

Figures (9)

• Figure 1: TokSuite is a comprehensive benchmark covering real-world perturbations that change tokenization (left), and 14 models that share the same initialization, architecture, and data but differ only in their tokenizers (right). Left panel illustrates how different tokenizers fragment the concept "doctor" when subjected to OCR errors, orthographic mistakes, semantic equivalents, emoji substitution, and multilingual translations. Each colored box represents one token across Gemma-2 (yellow), GPT-2 (blue), TokenMonster (green), and XGLM (red) tokenizers.
• Figure 2: Model Performance on Multilingual Benchmarks
• Figure 3: Tokenizer performance comparison across languages using Flores200 dataset with intrinsic efficiency metrics (fertility, parity, and proportion of continued words). Lower is better.
• Figure 4: Accuracies of models on canonical versus perturbed questions across the English (eng_Latn), Italian (ita_Latn), Farsi (pes_Arab), Turkish (tur_Latn), and Chinese (zho_Hans) TokSuite subsets.
• Figure 5: Left: Styling challenges that are normalized by NFKC, Right: Styling challenges that NFKC cannot