Random Text, Zipf's Law, Critical Length,and Implications for Large Language Models

TL;DR

The paper studies a deliberately minimal, non-linguistic text model in which independent symbol draws yield words as maximal non-space runs. It derives a geometric word-length distribution, closed-form expectations for word counts and distinct-word growth, and a Zipf-type rank–frequency law with exponent $\\alpha = 1 - \\frac{\\ln(1-q)}{\\ln m}$, all linked to segmentation and combinatorics. A central result is the critical word length $k^*$ separating a core, recurrent vocabulary from a tail of almost-surely unique words, providing a unified analytic bridge between word lengths, vocabulary growth, and rank-frequency structure. The authors argue this structural baseline clarifies which linguistic patterns require deeper explanation beyond random text and offers practical implications for interpreting token distributions in large language models and for designing structural benchmarks. Overall, the work presents a transparent, closed-form null model that captures several high-level statistical regularities without linguistic content, informing both quantitative linguistics and the evaluation of LLMs.

Abstract

We study a deliberately simple, fully non-linguistic model of text: a sequence of independent draws from a finite alphabet of letters plus a single space symbol. A word is defined as a maximal block of non-space symbols. Within this symbol-level framework, which assumes no morphology, syntax, or semantics, we derive several structural results. First, word lengths follow a geometric distribution governed solely by the probability of the space symbol. Second, the expected number of words of a given length, and the expected number of distinct words of that length, admit closed-form expressions based on a coupon-collector argument. This yields a critical word length k* at which word types transition from appearing many times on average to appearing at most once. Third, combining the exponential growth of the number of possible strings of length k with the exponential decay of the probability of each string, we obtain a Zipf-type rank-frequency law p(r) proportional to r^{-alpha}, with an exponent determined explicitly by the alphabet size and the space probability. Our contribution is twofold. Mathematically, we give a unified derivation linking word lengths, vocabulary growth, critical length, and rank-frequency structure in a single explicit model. Conceptually, we argue that this provides a structurally grounded null model for both natural-language word statistics and token statistics in large language models. The results show that Zipf-like patterns can arise purely from combinatorics and segmentation, without optimization principles or linguistic organization, and help clarify which phenomena require deeper explanation beyond random-text structure.