An Information-Theoretic Perspective on LLM Tokenizers
Mete Erdogan, Abhiram Gorle, Shubham Chandak, Mert Pilanci, Tsachy Weissman
TL;DR
This work reframes LLM tokenization through an information-theoretic lens, examining how tokenizer design—both pretrained GPT-family tokenizers and learned tokenizers—controls compression, induced short-range structure, and robustness to domain shift. By benchmarking tokenizers as compressors, studying entropy across training scales, and analyzing tokenization as a universal transform and a noiseless channel, the authors reveal consistent trends: more training data increases unigram entropy while lowering higher-order entropies, tokenizers can improve universal compression but may degrade under distribution shift, and a channel-centric perspective with capacity-utilization metrics provides a unifying view of the trade-offs. The paper also introduces a practical LZ-aware BPE approach and demonstrates measurable gains in compression performance, highlighting design principles for compression-aware tokenization. Overall, these insights offer principled guidance for developing tokenizers that balance efficiency, linguistic structure, and robustness in real-world, domain-diverse settings.
Abstract
Large language model (LLM) tokenizers act as structured compressors: by mapping text to discrete token sequences, they determine token count (and thus compute and context usage) and the statistical structure seen by downstream models. Despite their central role in LLM pipelines, the link between tokenization, compression efficiency and induced structure is not well understood. We empirically demonstrate that tokenizer training scale redistributes entropy: as training data grows, the token stream becomes more diverse in aggregate (higher unigram entropy) yet markedly more predictable in-context (lower higher-order conditional entropies), indicating that tokenization absorbs substantial short-range regularity although these gains degrade under train-test domain mismatch. To ground these observations, we first benchmark i) pretrained GPT-family tokenizers as black-box compressors across various domains, and ii) learned tokenizers across configurations spanning vocabulary size, training scale, and domain. Next, we study tokenization as a transform for universal compression and introduce a compression-aware BPE variant. Finally, we adopt a channel lens and introduce capacity-utilization metrics to analyze tokenizer behaviour and outline implications for downstream modeling. Put together, our results expose various trade-offs between compression, induced structure, and robustness under domain shift, and motivate principled, compression-aware tokenizer design.