Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs

Sachin Pawar, Manoj Apte, Kshitij Jadhav, Girish Keshav Palshikar, Nitin Ramrakhiyani

TL;DR

The paper addresses how tokenization fragmentation of natural words into multiple tokens can hurt LLM performance. It introduces tokenization penalty functions, notably the Contextual Penalty CP, which incorporates left-context, POS weighting, and model-specific next-token probabilities to quantify tokenization quality for texts. Across seven NLP tasks and four LLMs, CP demonstrates statistically significant effects on accuracy in a substantial fraction of dataset–model pairs, outperforming baseline penalties and illustrating the practical impact of tokenization design. The work also compares CP with perplexity, proposes mitigation strategies such as input transformations and morphology-aware tokenization, and highlights directions for broader evaluation and tokenizer improvements to minimize tokenization-induced degradation.

Abstract

Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model's fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of "natural" words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral's tokenizer splits "martial" into "mart" and "ial"). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs.

Broken Words, Broken Performance: Effect of Tokenization on Performance of LLMs

TL;DR

The paper addresses how tokenization fragmentation of natural words into multiple tokens can hurt LLM performance. It introduces tokenization penalty functions, notably the Contextual Penalty CP, which incorporates left-context, POS weighting, and model-specific next-token probabilities to quantify tokenization quality for texts. Across seven NLP tasks and four LLMs, CP demonstrates statistically significant effects on accuracy in a substantial fraction of dataset–model pairs, outperforming baseline penalties and illustrating the practical impact of tokenization design. The work also compares CP with perplexity, proposes mitigation strategies such as input transformations and morphology-aware tokenization, and highlights directions for broader evaluation and tokenizer improvements to minimize tokenization-induced degradation.

Abstract

Tokenization is the first step in training any Large Language Model (LLM), where the text is split into a sequence of tokens as per the model's fixed vocabulary. This tokenization in LLMs is different from the traditional tokenization in NLP where the text is split into a sequence of "natural" words. In LLMs, a natural word may also be broken into multiple tokens due to limited vocabulary size of the LLMs (e.g., Mistral's tokenizer splits "martial" into "mart" and "ial"). In this paper, we hypothesize that such breaking of natural words negatively impacts LLM performance on various NLP tasks. To quantify this effect, we propose a set of penalty functions that compute a tokenization penalty for a given text for a specific LLM, indicating how "bad" the tokenization is. We establish statistical significance of our hypothesis on multiple NLP tasks for a set of different LLMs.
Paper Structure (15 sections, 4 equations, 3 figures, 9 tables)

This paper contains 15 sections, 4 equations, 3 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Tokenization Penalty Functions
  3. Measuring Effect of Bad Tokenization
  4. Experiments
  5. NLP Tasks and Datasets
  6. Experimental Results
  7. Conclusions and Future Work
  8. NLP Tasks and Datasets
  9. Implementation Details
  10. Prompts used for the NLP tasks
  11. Examples of Penalty Functions
  12. Comparing Contextual Penalty (CP) with Perplexity
  13. Background concepts
  14. Isolation Forest (IF) liu2008isolation
  15. Byte Pair Encoding (BPE) sennrich2016neural

Figures (3)

  • Figure 1: Effect of splitting words into multiple tokens
  • Figure 2: Accuracy difference between top and bottom deciles of the instances as per CP (top_3) for Phi
  • Figure 3: Comparing LLMs across multiple aspects: fertility (average number of tokens per natural word across all the datasets ali2024tokenizer) and performance (number of datasets where the LLM is not significantly affected by tokenization, as per CP (top3) penalty function). The numbers in bracket after each LLM indicate its vocabulary size.