Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A path to natural language through tokenisation and transformers

David S. Berman, Alexander G. Stapleton

TL;DR

The paper investigates how Zipf's law and tokenisation depth interact in transformer-based language modelling. It derives a closed-form expression for the entropy of Zipfian corpora and shows that recursive BPE drives token frequencies toward Zipf's law, increasing empirical entropy and aligning with theoretical predictions. Across transformer experiments, softmax and sampled entropies converge toward Zipf-based expectations as vocabulary size grows, with attention analyses indicating diminishing local dependencies. Collectively, the results frame tokenisation as a principled statistical transform that preserves core informational properties and suggest a renormalisation-group perspective on language statistics.

Abstract

Natural languages exhibit striking regularities in their statistical structure, including notably the emergence of Zipf's and Heaps' laws. Despite this, it remains broadly unclear how these properties relate to the modern tokenisation schemes used in contemporary transformer models. In this note, we analyse the information content (as measured by the Shannon entropy) of various corpora under the assumption of a Zipfian frequency distribution, and derive a closed-form expression for the slot entropy expectation value. We then empirically investigate how byte--pair encoding (BPE) transforms corpus statistics, showing that recursive applications of BPE drive token frequencies toward a Zipfian power law while inducing a characteristic growth pattern in empirical entropy. Utilizing the ability of transformers to learn context dependent token probability distributions, we train language models on corpora tokenised at varying BPE depths, revealing that the model predictive entropies increasingly agree with Zipf-derived predictions as the BPE depth increases. Attention-based diagnostics further indicate that deeper tokenisation reduces local token dependencies, bringing the empirical distribution closer to the weakly dependent (near IID) regime. Together, these results clarify how BPE acts not only as a compression mechanism but also as a statistical transform that reconstructs key informational properties of natural language.

A path to natural language through tokenisation and transformers

TL;DR

The paper investigates how Zipf's law and tokenisation depth interact in transformer-based language modelling. It derives a closed-form expression for the entropy of Zipfian corpora and shows that recursive BPE drives token frequencies toward Zipf's law, increasing empirical entropy and aligning with theoretical predictions. Across transformer experiments, softmax and sampled entropies converge toward Zipf-based expectations as vocabulary size grows, with attention analyses indicating diminishing local dependencies. Collectively, the results frame tokenisation as a principled statistical transform that preserves core informational properties and suggest a renormalisation-group perspective on language statistics.

Abstract

Natural languages exhibit striking regularities in their statistical structure, including notably the emergence of Zipf's and Heaps' laws. Despite this, it remains broadly unclear how these properties relate to the modern tokenisation schemes used in contemporary transformer models. In this note, we analyse the information content (as measured by the Shannon entropy) of various corpora under the assumption of a Zipfian frequency distribution, and derive a closed-form expression for the slot entropy expectation value. We then empirically investigate how byte--pair encoding (BPE) transforms corpus statistics, showing that recursive applications of BPE drive token frequencies toward a Zipfian power law while inducing a characteristic growth pattern in empirical entropy. Utilizing the ability of transformers to learn context dependent token probability distributions, we train language models on corpora tokenised at varying BPE depths, revealing that the model predictive entropies increasingly agree with Zipf-derived predictions as the BPE depth increases. Attention-based diagnostics further indicate that deeper tokenisation reduces local token dependencies, bringing the empirical distribution closer to the weakly dependent (near IID) regime. Together, these results clarify how BPE acts not only as a compression mechanism but also as a statistical transform that reconstructs key informational properties of natural language.
Paper Structure (20 sections, 23 equations, 7 figures, 2 tables)

This paper contains 20 sections, 23 equations, 7 figures, 2 tables.

Table of Contents

  1. Introduction
  2. A primer of Zipf's law
  3. Encoding in transformers
  4. Entropy of Zipfian Corpora
  5. Theoretically predicted entropy of a Zipfian corpora
  6. Numerical demonstration
  7. Dependence of the entropy calculation on $s$
  8. Emergence of Zipf's law under byte-pair encoding (BPE)
  9. Numerical results
  10. Recursive applications of BPE lead to Zipf
  11. Empirical entropy under BPE
  12. An exploration with transformers
  13. Numerical setup
  14. Entropy of the Model's Softmax Distribution.
  15. Entropy of the Sampled Token Sequence.
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Zipf statistics across datasets.
  • Figure 2: Dependence of the empirical entropy calculation on $\text{rank}(s)$, the position of the fixed frequency for the 20Newsgroups dataset.
  • Figure 3: Fitted Zipf exponent under recursive BPE applications for the IMDB dataset.
  • Figure 4: Recursive applications of BPE on the IMDB review corpus lead to a corpus which is increasingly Zipfian.
  • Figure 5: Empirical and predicted entropies as a function of BPE recurse step $\iota$.
  • ...and 2 more figures