A Tale of Two Structures: Do LLMs Capture the Fractal Complexity of Language?

TL;DR

This work interrogates whether large language models can faithfully reproduce the fractal structure of natural language, quantified by the Hölder exponent $S$ and the Hurst exponent $H$, and how factors such as decoding temperature and prompting influence this capability. By generating extensive corpora across pretrained and instruction-tuned models and a public GAGLE dataset, the authors show that natural language occupies a relatively narrow fractal-parameter range, while LLM outputs span a much wider spectrum, with larger models aligning fractal properties more closely to NL but still exhibiting biases toward higher $S$ and $H$. They demonstrate that $H$ is a strong predictor of output quality and that prompting often has a dominant, non-monotone effect on fractal measures, including a double-descent in self-similarity when adding contextual prompts. The study reveals partial fractal mimicry by LLMs, robust across architectures, and suggests that fractal parameters, in combination with first-order metrics, could aid in detecting synthetic texts and guiding generation and evaluation strategies. The work thus contributes to understanding the complex interplay between fractal language properties, prompting, and statistical mimicry in LLMs, with public datasets to spur further research in detection and evaluation.

Abstract

Language exhibits a fractal structure in its information-theoretic complexity (i.e. bits per token), with self-similarity across scales and long-range dependence (LRD). In this work, we investigate whether large language models (LLMs) can replicate such fractal characteristics and identify conditions-such as temperature setting and prompting method-under which they may fail. Moreover, we find that the fractal parameters observed in natural language are contained within a narrow range, whereas those of LLMs' output vary widely, suggesting that fractal parameters might prove helpful in detecting a non-trivial portion of LLM-generated texts. Notably, these findings, and many others reported in this work, are robust to the choice of the architecture; e.g. Gemini 1.0 Pro, Mistral-7B and Gemma-2B. We also release a dataset comprising of over 240,000 articles generated by various LLMs (both pretrained and instruction-tuned) with different decoding temperatures and prompting methods, along with their corresponding human-generated texts. We hope that this work highlights the complex interplay between fractal properties, prompting, and statistical mimicry in LLMs, offering insights for generating, evaluating and detecting synthetic texts.

Figures (35)

• Figure 1: A causal model we consider, in which a latent "context" generates a prefix and both produce a suffix.
• Figure 2: Quality of power law fit for fractal parameters in both human- and LLM-generated documents, for the same setting as in Appendix \ref{['sect:app:rho_doc_sample']} where samples of documents are provided.
• Figure 3: The $y$-axis is either $\log\tilde{\mathrm{S}}/\mathrm{S}$ (left column) or $\log\tilde{\mathrm{H}}/\mathrm{H}$ (right column), where $\tilde{\mathrm{S}}$ is the Hölder exponent of LLM-generated texts while $\mathrm{S}$ is of natural language, and the same holds for $\tilde{\mathrm{H}}$ and $\mathrm{H}$. The $x$-axis are the generating models: Gemini 1.0 Pro (denoted G-P), Mistral-7B (denoted M-7), and Gemma-2B (denoted G-2), all are pretrained models with temperature $\beta=1$. Subtitles indicate the model used for scoring the texts. As expected, we observe that larger models tend to replicate the fractal properties of natural language better than smaller models. In addition, LLM-generated texts has higher values of both $\mathrm{S}$ (less self-similarity) and $\mathrm{H}$ (more dependence).
• Figure 4: $y$-axis is the log-ratio of log-PPL scores for both pretrained (PT) and instruction-tuned (IT) models with simple continuation, when Gemini 1.0 Pro (left), Mistral-7B (center), and Gemma-2B (right) is used to score texts. Texts generated by large pretrained models do not have a lower log-PPL than natural texts; instruction tuning and the use of small temperatures lead to that effect.
• Figure 5: Output of the GLTR tool gehrmann-etal-2019-gltr on texts generated by humans (left), Mistral-7B pretrained (middle) and Mistral-7B instruction-tuned (right) at temperature $\beta=1.0$. Both the pretrained and instruction-tuned models are provided with a short prefix. Colors indicate perplexity scores. The output of the pretrained model looks similar to the human-generated text in terms of log-PPL scores (more orange, red, and purple tokens indicating surprise), in agreement with Figure \ref{['fig:ppl_gemini']}.