Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Two Counterexamples to Tokenization and the Noiseless Channel

Marco Cognetta, Vilém Zouhar, Sangwhan Moon, Naoaki Okazaki

TL;DR

This work challenges the claim that Rényi efficiency of the unigram distribution is a reliable intrinsic predictor of downstream NLP performance. It introduces two counterexample families for BPE tokenization—Random-Drop BPE and Duplication BPE—that can arbitrarily increase Rényi efficiency while decreasing BLEU in machine translation. Through theoretical arguments and large-scale MT experiments, the authors show that Rényi efficiency can fail to predict downstream outcomes under certain tokenization perturbations, while other intrinsic metrics (PCT, SEQ) may better signal degradation in some cases. The findings suggest refining tokenization predictors to account for vocabulary growth and token-level perturbations, ultimately guiding the development of more robust intrinsic metrics for tokenizer evaluation.

Abstract

In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase Rényi efficiency while decreasing the downstream model performance. These counterexamples expose cases where Rényi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.

Two Counterexamples to Tokenization and the Noiseless Channel

TL;DR

This work challenges the claim that Rényi efficiency of the unigram distribution is a reliable intrinsic predictor of downstream NLP performance. It introduces two counterexample families for BPE tokenization—Random-Drop BPE and Duplication BPE—that can arbitrarily increase Rényi efficiency while decreasing BLEU in machine translation. Through theoretical arguments and large-scale MT experiments, the authors show that Rényi efficiency can fail to predict downstream outcomes under certain tokenization perturbations, while other intrinsic metrics (PCT, SEQ) may better signal degradation in some cases. The findings suggest refining tokenization predictors to account for vocabulary growth and token-level perturbations, ultimately guiding the development of more robust intrinsic metrics for tokenizer evaluation.

Abstract

In Tokenization and the Noiseless Channel (Zouhar et al., 2023a), Rényi efficiency is suggested as an intrinsic mechanism for evaluating a tokenizer: for NLP tasks, the tokenizer which leads to the highest Rényi efficiency of the unigram distribution should be chosen. The Rényi efficiency is thus treated as a predictor of downstream performance (e.g., predicting BLEU for a machine translation task), without the expensive step of training multiple models with different tokenizers. Although useful, the predictive power of this metric is not perfect, and the authors note there are additional qualities of a good tokenization scheme that Rényi efficiency alone cannot capture. We describe two variants of BPE tokenization which can arbitrarily increase Rényi efficiency while decreasing the downstream model performance. These counterexamples expose cases where Rényi efficiency fails as an intrinsic tokenization metric and thus give insight for building more accurate predictors.
Paper Structure (23 sections, 3 theorems, 19 equations, 2 figures, 5 tables, 1 algorithm)

This paper contains 23 sections, 3 theorems, 19 equations, 2 figures, 5 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Byte-Pair Encoding
  3. BPE Training.
  4. BPE Tokenization.
  5. Rényi Efficiency and Other Metrics
  6. Baseline.
  7. Percentile Frequency.
  8. Rényi Efficiency.
  9. Counterexamples
  10. Random-Drop BPE
  11. Efficiency Increase.
  12. Duplication BPE
  13. Explanation.
  14. Rényi Entropy.
  15. Rényi Efficiency.
  16. ...and 8 more sections

Key Result

Theorem 4.1

Let $c_{\mathcal{V}}(w)$ be the unigram frequency of $w{\in} \mathcal{V}$, and $T_{\mathcal{V}} = \sum_{w \in \mathcal{V}} c(w)$ so that $p_{\mathcal{V}}(w) = \frac{c_{\mathcal{V}}(w)}{T_{\mathcal{V}}}$ is the unigram distribution. Suppose subword $x = (y, z) \in \mathcal{V}$ is chosen for decomposi

Figures (2)

  • Figure 1: The same model is trained on two different tokenizations. We are looking for metric such that, across multiple settings, score 1 > score 2 if and only if performance 1 > performance 2.
  • Figure 2: An example of Random-Drop BPE with $\mathcal{D} = \{\texttt{ENCOD},~\texttt{COD}\}$. The tokenization process is two-staged: (1) a regular BPE tokenizer is used to produce a tokenization, and (2) Random-Drop BPE recursively decomposes subwords that appear in the set $\mathcal{D}$ (first $\texttt{ENCOD}$, and then $\texttt{COD}$). Since $\texttt{ING}$ is not in $\mathcal{D}$, it is unchanged during the decomposition step. The final tokenized text is EN -CO -D -ING. If $\mathcal{D}$ was just $\{\texttt{ENCOD}\}$, then the final tokenization would have been EN -COD -ING, and if it was just $\{\texttt{COD}\}$, the final tokenization would have been $\texttt{ENCOD -ING}$, since $\texttt{-COD}$ was not in the final tokenization produced by the initial BPE pass.

Theorems & Definitions (3)

  • Theorem 4.1
  • Theorem 4.2
  • Theorem 4.3