Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

Sawsan Alqahtani, Mir Tafseer Nayeem, Md Tahmid Rahman Laskar, Tasnim Mohiuddin, M Saiful Bari

TL;DR

The paper reframes tokenization from a preprocessing step to a core design decision that shapes model efficiency, fairness, and linguistic faithfulness. It advocates a context-aware, tokenizer–model co-design framework with standardized evaluation across dimensions like coverage, generalizability, linguistic alignment, robustness, and representation utilization. A structured iterative process for tokenizer development is proposed, including decisions on training from scratch, adaptation, and responsible design, complemented by a multi-dimensional evaluation framework and transparent reporting. The work highlights practical pathways to reduce token-length disparities, improve cross-language parity, and enable fairer, more efficient LLM deployment across languages and domains, while acknowledging that empirical validation and tooling are still required to operationalize these ideas.

Abstract

Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable.

Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

TL;DR

The paper reframes tokenization from a preprocessing step to a core design decision that shapes model efficiency, fairness, and linguistic faithfulness. It advocates a context-aware, tokenizer–model co-design framework with standardized evaluation across dimensions like coverage, generalizability, linguistic alignment, robustness, and representation utilization. A structured iterative process for tokenizer development is proposed, including decisions on training from scratch, adaptation, and responsible design, complemented by a multi-dimensional evaluation framework and transparent reporting. The work highlights practical pathways to reduce token-length disparities, improve cross-language parity, and enable fairer, more efficient LLM deployment across languages and domains, while acknowledging that empirical validation and tooling are still required to operationalize these ideas.

Abstract

Tokenization underlies every large language model, yet it remains an under-theorized and inconsistently designed component. Common subword approaches such as Byte Pair Encoding (BPE) offer scalability but often misalign with linguistic structure, amplify bias, and waste capacity across languages and domains. This paper reframes tokenization as a core modeling decision rather than a preprocessing step. We argue for a context-aware framework that integrates tokenizer and model co-design, guided by linguistic, domain, and deployment considerations. Standardized evaluation and transparent reporting are essential to make tokenization choices accountable and comparable. Treating tokenization as a core design problem, not a technical afterthought, can yield language technologies that are fairer, more efficient, and more adaptable.
Paper Structure (45 sections, 2 figures, 7 tables)

This paper contains 45 sections, 2 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Our Perspective on Tokenization
  3. Alternative Perspectives on Tokenization
  4. Tokenization and Alternatives
  5. Subword Tokenization Methods
  6. Emerging Alternatives for the De Facto Tokenization Paradigm
  7. Learning Compositional Structure Explicitly:
  8. From Subwords to Raw Bytes:
  9. Modeling Language at Semantic Level:
  10. Rethinking Tokenizer Design: From Pitfalls to Practice
  11. Design Integration:
  12. Adaptation and Generalization:
  13. Evaluation Practices, Reliability, and Bias:
  14. A Structured Iterative Process
  15. Critical Decisions within the Workflow
  16. ...and 30 more sections

Figures (2)

  • Figure 1: Thematic comparison of current practices and proposed design principles for tokenizer development.
  • Figure 2: Tokenization of the Arabic word using different tokenization strategies.