Fractal Patterns May Illuminate the Success of Next-Token Prediction

TL;DR

The study models language as a self-similar, long-range dependent process and introduces fractal descriptors—self-similarity exponent $\mathrm{S}$, Hurst parameter $\mathrm{H}$, fractal dimension $\mathrm{D}$, and Joseph exponent $\mathrm{J}$—to quantify intrinsic linguistic complexity. Using bits-per-byte and token-probability-derived increments $z_t=-\log p(w_t|w_{[t-1]})$, applied to up to $2048$-token prefixes across multiple domains and models (e.g., PaLM/PaLM2, T5), the paper reports robust medians $\mathrm{S}=0.59\pm0.08$, $\mathrm{H}=0.70\pm0.09$, $\mathrm{D}=1.41\pm0.08$, and $\mathrm{J}=0.49\pm0.08$, indicating strong self-similarity and long-range dependence in language. A key finding is that model-specific variations in $\mathrm{H}$ correlate with downstream performance beyond BPB, and a combined predictor $\mathrm{H}_B=1/\mathrm{BPB}+\mathrm{H}$ yields higher $R^2$ (up to ~0.86) than BPB alone, suggesting fractal metrics capture actionable predictive information. A negative result shows that longer training-context during pretraining did not improve downstream results, implying that fractal structure, rather than mere context length, underpins multiscale language organization and intelligence-like behavior. The work highlights a new lens for understanding LLM capabilities and motivates future cross-domain validation and exploration of fractal-guided optimization strategies.

Abstract

We study the fractal structure of language, aiming to provide a precise formalism for quantifying properties that may have been previously suspected but not formally shown. We establish that language is: (1) self-similar, exhibiting complexities at all levels of granularity, with no particular characteristic context length, and (2) long-range dependent (LRD), with a Hurst parameter of approximately H=0.7. Based on these findings, we argue that short-term patterns/dependencies in language, such as in paragraphs, mirror the patterns/dependencies over larger scopes, like entire documents. This may shed some light on how next-token prediction can capture the structure of text across multiple levels of granularity, from words and clauses to broader contexts and intents. In addition, we carry out an extensive analysis across different domains and architectures, showing that fractal parameters are robust. Finally, we demonstrate that the tiny variations in fractal parameters seen across LLMs improve upon perplexity-based bits-per-byte (BPB) in predicting their downstream performance. We hope these findings offer a fresh perspective on language and the mechanisms underlying the success of LLMs.

Figures (7)

• Figure 1: Manifestations of processes across different time scales. A region marked in red corresponds to the magnified plot beneath it. left: The process exhibits self-similarity with rich details at all levels of granularity. It is an integral process $(X_t)_{t\in\mathbb{N}}$ calculated from Wikipedia (see Section \ref{['sect:defs']}). right: Example of a process that is not self-similar, looking smoother at larger time scales.
• Figure 2: Peak probability $p_\epsilon(\tau)$ is plotted against the granularity level $\tau$ (see Section \ref{['sect:fract:selfsim']}). We observe power laws $p_\epsilon(\tau)\sim\tau^{-\mathrm{S}}$, indicating self-similarity, with a median exponent of $\mathrm{S}=0.59\pm0.08$.
• Figure 3: Rescaled range $R(n)/S(n)$ is plotted against the number of normalized bits $n$. We observe a power law $R(n)/S(n)\sim n^\mathrm{H}$ in all domains. When aggregating all datasets, $\mathrm{H}=0.70\pm0.09$.
• Figure 4: left: Estimates of the self-similarity exponent $\mathrm{S}$ are generally robust to the choice of $\epsilon$. right: The partial auto-correlation function calculated across domains. DM Mathematics has a much shorter dependence compared to the rest of the domains, in agreement with its Hurst parameter.
• Figure 5: The standard deviation $\sigma$ of the $\tau$-increments $X_{t+\tau}-X_t$ is plotted against the scale $\tau$. We, again, observe another power law relation $\sigma\sim \tau^\mathrm{J}$, with a Joseph exponent $\mathrm{J}=0.49\pm0.08$.