Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance

TL;DR

There is a correlation between tokenizers' compression and models' downstream performance, suggesting that compression is a reliable intrinsic indicator of tokenization quality.

Abstract

Despite it being the cornerstone of BPE, the most common tokenization algorithm, the importance of compression in the tokenization process is still unclear. In this paper, we argue for the theoretical importance of compression, that can be viewed as 0-gram language modeling where equal probability is assigned to all tokens. We also demonstrate the empirical importance of compression for downstream success of pre-trained language models. We control the compression ability of several BPE tokenizers by varying the amount of documents available during their training: from 1 million documents to a character-based tokenizer equivalent to no training data at all. We then pre-train English language models based on those tokenizers and fine-tune them over several tasks. We show that there is a correlation between tokenizers' compression and models' downstream performance, suggesting that compression is a reliable intrinsic indicator of tokenization quality. These correlations are more pronounced for generation tasks (over classification) or for smaller models (over large ones). We replicated a representative part of our experiments on Turkish and found similar results, confirming that our results hold for languages with typological characteristics dissimilar to English. We conclude that building better compressing tokenizers is a fruitful avenue for further research and for improving overall model performance.

Figures (6)

• Figure 1: Generation performance of the various models averaged over both generation tasks. For each model size the results are presented as relative compared to the 1m-doc model.
• Figure 2: Six tokenizers differing in the amount of supporting documents tokenizing the same sentence. Note that better compression is achieved with more support.
• Figure 3: Number of subwords per English word as a function of its abundance in 3 million unseen documents. Averaged over orders of magnitude. The number words included in each bin is indicated under the x axis.
• Figure 4: Number of subwords per Turkish word as a function of its abundance in 3 million unseen documents. Averaged over orders of magnitude. The number words included in each bin is indicated under the x axis.
• Figure 5: Downstream success in Rouge-L relative to the 1m-doc model plotted against the average frequency in each example. Trend lines were plotted based on the entire data, but for visibility reasons the scatter is based on averages over bins containing each 2% of data.