Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Reducing Tokenization Premiums for Low-Resource Languages

Geoffrey Churchill, Steven Skiena

TL;DR

Tokenization premiums cause low-resource languages to be encoded with many more tokens than English, inflating costs and truncating context. The paper analyzes ten tokenizers and FLORES-200 to quantify premiums, revealing language- and script-dependent disparities, with non-Latin scripts often paying substantial penalties. To mitigate this, it proposes post-hoc vocabulary augmentation by adding per-character tokens and evaluates embedding strategies ($k$NN, Linear regression, Local linear regression) to map new tokens into the frozen model’s embedding space, achieving high preservation of representations in many languages. The findings suggest a practical path to cheaper, more inclusive multilingual LM deployment, while highlighting limitations and avenues for future vocabulary-design and embedding-space approaches.

Abstract

Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs, meaning that it generally requires several times as many tokens to encode a sentence in a low-resource language than to encode the analogous sentence in English. This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages. In this paper we analyze the tokenizers of ten popular LMs to better understand their designs and per-language tokenization premiums. We also propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens. We apply this methodology to 12 low-resource languages, demonstrating that the original and compressed inputs often have similar last hidden states when run through the Llama 3.2 1B model.

Reducing Tokenization Premiums for Low-Resource Languages

TL;DR

Tokenization premiums cause low-resource languages to be encoded with many more tokens than English, inflating costs and truncating context. The paper analyzes ten tokenizers and FLORES-200 to quantify premiums, revealing language- and script-dependent disparities, with non-Latin scripts often paying substantial penalties. To mitigate this, it proposes post-hoc vocabulary augmentation by adding per-character tokens and evaluates embedding strategies (NN, Linear regression, Local linear regression) to map new tokens into the frozen model’s embedding space, achieving high preservation of representations in many languages. The findings suggest a practical path to cheaper, more inclusive multilingual LM deployment, while highlighting limitations and avenues for future vocabulary-design and embedding-space approaches.

Abstract

Relative to English, low-resource languages suffer from substantial tokenization premiums in modern LMs, meaning that it generally requires several times as many tokens to encode a sentence in a low-resource language than to encode the analogous sentence in English. This tokenization premium results in increased API and energy costs and reduced effective context windows for these languages. In this paper we analyze the tokenizers of ten popular LMs to better understand their designs and per-language tokenization premiums. We also propose a mechanism to reduce tokenization premiums in pre-trained models, by post-hoc additions to the token vocabulary that coalesce multi-token characters into single tokens. We apply this methodology to 12 low-resource languages, demonstrating that the original and compressed inputs often have similar last hidden states when run through the Llama 3.2 1B model.
Paper Structure (16 sections, 2 equations, 3 figures, 4 tables)

This paper contains 16 sections, 2 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Tokenization:
  4. Frozen Model Adaptation:
  5. Comparing Tokenizers
  6. Vocabulary Comparisons
  7. The FLORES-200 dataset
  8. Language-Specific Tokenization Premiums
  9. Mitigating Tokenization Premiums in Pre-Trained Models
  10. Mitigation Methods
  11. $k$-nearest neighbors:
  12. Linear regression:
  13. Local linear regression:
  14. Experimental Results
  15. Conclusions and Future Directions
  16. ...and 1 more sections

Figures (3)

  • Figure 1: Jaccard similarities between normalized tokenizer vocabularies.
  • Figure 2: Tokenization premiums of several languages by number of speakers, averaged over all tokenizers. Some language codes, like "arb" and "zho", are available in two scripts and so appear twice in the plot.
  • Figure 3: Fraction of smaller normalized vocabulary included in larger normalized vocabulary.