How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers

TL;DR

This paper addresses the instability of token counts as a universal length unit for frontier LLMs by performing an empirical analysis of tokenization across eight domains and ten tokenizers. The authors quantify character-to-token compression with $c = \dfrac{n_{chars}}{n_{tokens}}$, and examine how tokenization affects context limits and pricing, revealing substantial cross-domain and cross-tokenizer variation. They challenge common heuristics, such as the 0.75 words/token rule, and show that model-native context limits are not directly comparable across domains or models. The work provides practical guidance for interpreting token-based metrics, enabling more accurate benchmarking, cost estimation, and fair evaluation of LLM capabilities across diverse text types.

Abstract

Frontier LLMs are increasingly utilised across academia, society and industry. A commonly used unit for comparing models, their inputs and outputs, and estimating inference pricing is the token. In general, tokens are used as a stable currency, assumed to be broadly consistent across tokenizers and contexts, enabling direct comparisons. However, tokenization varies significantly across models and domains of text, making naive interpretation of token counts problematic. We quantify this variation by providing a comprehensive empirical analysis of tokenization, exploring the compression of sequences to tokens across different distributions of textual data. Our analysis challenges commonly held heuristics about token lengths, finding them to be overly simplistic. We hope the insights of our study add clarity and intuition toward tokenization in contemporary LLMs.

Figures (13)

• Figure 1: Tokenization schemes vary significantly. Token boundaries are represented as shaded colours and token counts are shown in purple. Antidisestablishmentarianism is tokenized many different ways, using between 4 (Fuyu) and 9 tokens (early Llama models). Variation in tokenization is also observable for words containing fewer unique letters and more vowels, such as the Hawaiian word humuhumunukunukuāpua'a (Reef Triggerfish). Small changes to tokenizers within model families result in different tokenization, even in relatively simple sentences. $^*$From berlin2013hedgehog.
• Figure 2: Mean tokenizer compression ratio across different text domains. Error bars are calculated using the standard error over 50 deterministic samplings (each $\ge$1000 characters).
• Figure 3: Compression ratios of essays graham_essays_web translated into different languages. Compression ratios are averaged over 11 essays; common crawl prevalence is estimated from abadji-etal-2022-towards. Where necessary, arbitrary offsets have been added to the x-coordinates to reduce overlap between languages.
• Figure 4: Word to token compression variation for frequency-ranked English words. Shaded regions show 95% confidence intervals. Mean words/token for randomly selected English words are far lower (0.35--0.45).
• Figure 5: Token-character mapping varies significantly across domains and tokenizers. Lines are plotted using the inverse of the compression ratio as the gradient: steeper slopes require more tokens per character. Shaded regions show 95% confidence intervals.