Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiency

Hailay Kidu Teklehaymanot, Wolfgang Nejdl

TL;DR

Tokenization disparities are presented as a form of infrastructure bias that shapes access to multilingual AI across 200+ languages. Using FLORES-200 with Unicode NFC normalization and the cl100kbase tiktoken tokenizer, the study quantifies Tokens Per Sentence (TPS), Characters Per Token (CPT), and Relative Tokenization Cost (RTC) to reveal pronounced efficiency gaps between Latin-script and non-Latin/morphologically complex languages. The results show RTC can reach over four times the English baseline for some languages, with Latin-script languages achieving far more favorable tokenization profiles than non-Latin scripts. The work argues for linguistically informed tokenization strategies and adaptive vocabulary construction to reduce inequities and improve computational efficiency and accessibility in multilingual LLM systems.

Abstract

Tokenization disparities pose a significant barrier to achieving equitable access to artificial intelligence across linguistically diverse populations. This study conducts a large-scale cross-linguistic evaluation of tokenization efficiency in over 200 languages to systematically quantify computational inequities in large language models (LLMs). Using a standardized experimental framework, we applied consistent preprocessing and normalization protocols, followed by uniform tokenization through the tiktoken library across all language samples. Comprehensive tokenization statistics were collected using established evaluation metrics, including Tokens Per Sentence (TPS) and Relative Tokenization Cost (RTC), benchmarked against English baselines. Our cross-linguistic analysis reveals substantial and systematic disparities: Latin-script languages consistently exhibit higher tokenization efficiency, while non-Latin and morphologically complex languages incur significantly greater token inflation, often 3-5 times higher RTC ratios. These inefficiencies translate into increased computational costs and reduced effective context utilization for underrepresented languages. Overall, the findings highlight structural inequities in current AI systems, where speakers of low-resource and non-Latin languages face disproportionate computational disadvantages. Future research should prioritize the development of linguistically informed tokenization strategies and adaptive vocabulary construction methods that incorporate typological diversity, ensuring more inclusive and computationally equitable multilingual AI systems.

Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiency

TL;DR

Tokenization disparities are presented as a form of infrastructure bias that shapes access to multilingual AI across 200+ languages. Using FLORES-200 with Unicode NFC normalization and the cl100kbase tiktoken tokenizer, the study quantifies Tokens Per Sentence (TPS), Characters Per Token (CPT), and Relative Tokenization Cost (RTC) to reveal pronounced efficiency gaps between Latin-script and non-Latin/morphologically complex languages. The results show RTC can reach over four times the English baseline for some languages, with Latin-script languages achieving far more favorable tokenization profiles than non-Latin scripts. The work argues for linguistically informed tokenization strategies and adaptive vocabulary construction to reduce inequities and improve computational efficiency and accessibility in multilingual LLM systems.

Abstract

Tokenization disparities pose a significant barrier to achieving equitable access to artificial intelligence across linguistically diverse populations. This study conducts a large-scale cross-linguistic evaluation of tokenization efficiency in over 200 languages to systematically quantify computational inequities in large language models (LLMs). Using a standardized experimental framework, we applied consistent preprocessing and normalization protocols, followed by uniform tokenization through the tiktoken library across all language samples. Comprehensive tokenization statistics were collected using established evaluation metrics, including Tokens Per Sentence (TPS) and Relative Tokenization Cost (RTC), benchmarked against English baselines. Our cross-linguistic analysis reveals substantial and systematic disparities: Latin-script languages consistently exhibit higher tokenization efficiency, while non-Latin and morphologically complex languages incur significantly greater token inflation, often 3-5 times higher RTC ratios. These inefficiencies translate into increased computational costs and reduced effective context utilization for underrepresented languages. Overall, the findings highlight structural inequities in current AI systems, where speakers of low-resource and non-Latin languages face disproportionate computational disadvantages. Future research should prioritize the development of linguistically informed tokenization strategies and adaptive vocabulary construction methods that incorporate typological diversity, ensuring more inclusive and computationally equitable multilingual AI systems.
Paper Structure (24 sections, 3 equations, 4 figures)

This paper contains 24 sections, 3 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background
  4. Methodology
  5. Dataset
  6. Tokenization Model
  7. Experimental Procedure
  8. Preprocessing and Normalization
  9. Tokenization with tiktoken
  10. Tokenization Statistics Extraction
  11. Cross-Linguistic Efficiency Analysis
  12. Tokenizer Evaluation Metrics
  13. Tokens Per Sentence (TPS)
  14. Characters Per Token (CPT)
  15. Relative Tokenization Cost (RTC)
  16. ...and 9 more sections

Figures (4)

  • Figure 1: Comparison of scripts used by multiple languages. The bar chart shows the number of languages using each script, with Latin (Latn) being the most widely adopted. An annotation box lists scripts used by only a single language, such as Armenian (Armn), Georgian (Geor), and Japanese (Jpan). This figure highlights the distribution and concentration of script usage across languages.
  • Figure 2: Tokenization Efficiency Across Writing Scripts (Tokens Per Sentence - TPS). Cross-script comparison quantifies extreme tokenization burden variations, with the Myanmar script requiring the highest token density (357.2 TPS) and the Latin script achieving optimal efficiency (50.2 TPS). The language-wide mean of 90.3 TPS serves as a reference, revealing that numerous scripts require 3-4 times more tokens than average, with some scripts demanding nearly 7-fold higher computational costs for equivalent semantic content.
  • Figure 3: Average Characters Per Token (CPT) by Writing Script. Script-based analysis reveals dramatic efficiency variations, with Latin script achieving optimal compression (2.61 CPT) compared to numerous Asian scripts falling below 1.0 CPT. Non-Latin scripts consistently demonstrate lower efficiency, with Tibetan (0.49 CPT), Oriya (0.40 CPT), and Ol Chiki (0.41 CPT) exhibiting severe tokenization inefficiencies. This pattern reflects the Latin-script optimization inherent in contemporary tokenization algorithms.
  • Figure 4: Average Characters Per Token (CPT) by Language Family. The analysis reveals substantial variation across language families, with Creole languages demonstrating the highest tokenization efficiency (2.85 CPT) and Kannada exhibiting the lowest efficiency (0.55 CPT). Higher CPT values indicate more efficient character-to-token compression, while lower values reflect excessive subword fragmentation. Notable patterns include the superior performance of Creole, Isolate, and Austronesian families compared to morphologically complex families such as Dravidian and Tai-Kadai.