Large language models and the entropy of English

TL;DR

The distribution of code lengths reveals an emergent certainty about an increasing fraction of characters at large $N$, suggesting that long-ranged structure is learned only gradually and constrain efforts to build statistical physics models of LLMs or language itself.

Abstract

We use large language models (LLMs) to uncover long-ranged structure in English texts from a variety of sources. The conditional entropy or code length in many cases continues to decrease with context length at least to $N\sim 10^4$ characters, implying that there are direct dependencies or interactions across these distances. A corollary is that there are small but significant correlations between characters at these separations, as we show from the data independent of models. The distribution of code lengths reveals an emergent certainty about an increasing fraction of characters at large $N$. Over the course of model training, we observe different dynamics at long and short context lengths, suggesting that long-ranged structure is learned only gradually. Our results constrain efforts to build statistical physics models of LLMs or language itself.

Figures (6)

• Figure 1: Code length vs. context length across models. We evaluate $L(N)$ from Eq (\ref{['Ldef']}) for the C4 corpus using the OLMo 2 1B olmo20242olmo2furious, Llama 3.2 1B grattafiori2024llama3herdmodels, Qwen3 8B qwen3technicalreport, and DCLM 1.7B li2024datacomplm models. While there are differences of detail, all of these well trained models yield remarkably similar results. Error bars computed from the variance across random subsets of the data are smaller than the "hash" from point--to--point variability.
• Figure 2: Code length across genres. We evaluate $L(N)$ from Eq. (\ref{['Ldef']}) via the OLMo 2 1B model over three text corpora: the C4 internet corpus (as in Fig. \ref{['fig01']}), English Wikipedia wikidump, and the Gutenberg Poetry Corpus parrish_gutenberg_poetry_corpus.
• Figure 3: Distribution of conditional entropy and code length. The distribution of conditional entropy (a) and code length (b) across text samples evolves with the context length $K$. Data from the C4 corpus as seen through the OLMo 2 1B model as in Fig. \ref{['fig02']}. Results here are per token rather than per character; the OLMo 2 1B tokenizer uses 100,278 distinct tokens, with a mean number of characters per token $N/K = 4.79$ on the C4 dataset. Inset in (b) shows a log--log plot of the distribution, highlighting the near power--law tail at $\ell \rightarrow 0$.
• Figure 4: Development of code lengths during learning. Results for the DCLM 1.7B model li2024datacomplm, at approximately equal intervals of training toward the final model. Note the greater logarithmic decrease in code lengths at large context length. Error bars are smaller than the hash, as in Fig \ref{['fig01']}.
• Figure 5: Mutual information between characters as a function of separation. We show $I(d)$ from Eq. (\ref{['Id_def']}) for the C4 and English Wikipedia corpora. Errors computed from the variance across fractions of the data are smaller than the symbols.