Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Correlation Dimension of Auto-Regressive Large Language Models

Xin Du, Kumiko Tanaka-Ishii

TL;DR

This work introduces correlation dimension as a fractal-geometric metric to quantify long-range textual structure as perceived by autoregressive LLMs, addressing the limitations of perplexity in capturing global coherence. By analyzing sequences of next-token log-probability vectors via a correlation integral, it defines a dimension $d$ that reflects the recurrence structure of language, with empirical convergence to about $d\approx6.5$ across well-trained models and languages. The study reveals a three-stage pretraining evolution, sensitivity to contextual constraints, and the ability to detect degeneration and hallucination beyond local measures, while demonstrating robustness to quantization and applicability across architectures. The proposed metric offers a practical online signal for training diagnostics and generation control, providing deeper insight into generative dynamics and long-range dependencies in LLMs.

Abstract

Large language models (LLMs) have achieved remarkable progress in natural language generation, yet they continue to display puzzling behaviors -- such as repetition and incoherence -- even when exhibiting low perplexity. This highlights a key limitation of conventional evaluation metrics, which emphasize local prediction accuracy while overlooking long-range structural complexity. We introduce correlation dimension, a fractal-geometric measure of self-similarity, to quantify the epistemological complexity of text as perceived by a language model. This measure captures the hierarchical recurrence structure of language, bridging local and global properties in a unified framework. Through extensive experiments, we show that correlation dimension (1) reveals three distinct phases during pretraining, (2) reflects context-dependent complexity, (3) indicates a model's tendency toward hallucination, and (4) reliably detects multiple forms of degeneration in generated text. The method is computationally efficient, robust to model quantization (down to 4-bit precision), broadly applicable across autoregressive architectures (e.g., Transformer and Mamba), and provides fresh insight into the generative dynamics of LLMs.

Correlation Dimension of Auto-Regressive Large Language Models

TL;DR

This work introduces correlation dimension as a fractal-geometric metric to quantify long-range textual structure as perceived by autoregressive LLMs, addressing the limitations of perplexity in capturing global coherence. By analyzing sequences of next-token log-probability vectors via a correlation integral, it defines a dimension that reflects the recurrence structure of language, with empirical convergence to about across well-trained models and languages. The study reveals a three-stage pretraining evolution, sensitivity to contextual constraints, and the ability to detect degeneration and hallucination beyond local measures, while demonstrating robustness to quantization and applicability across architectures. The proposed metric offers a practical online signal for training diagnostics and generation control, providing deeper insight into generative dynamics and long-range dependencies in LLMs.

Abstract

Large language models (LLMs) have achieved remarkable progress in natural language generation, yet they continue to display puzzling behaviors -- such as repetition and incoherence -- even when exhibiting low perplexity. This highlights a key limitation of conventional evaluation metrics, which emphasize local prediction accuracy while overlooking long-range structural complexity. We introduce correlation dimension, a fractal-geometric measure of self-similarity, to quantify the epistemological complexity of text as perceived by a language model. This measure captures the hierarchical recurrence structure of language, bridging local and global properties in a unified framework. Through extensive experiments, we show that correlation dimension (1) reveals three distinct phases during pretraining, (2) reflects context-dependent complexity, (3) indicates a model's tendency toward hallucination, and (4) reliably detects multiple forms of degeneration in generated text. The method is computationally efficient, robust to model quantization (down to 4-bit precision), broadly applicable across autoregressive architectures (e.g., Transformer and Mamba), and provides fresh insight into the generative dynamics of LLMs.
Paper Structure (44 sections, 1 theorem, 6 equations, 8 figures, 14 tables, 1 algorithm)

This paper contains 44 sections, 1 theorem, 6 equations, 8 figures, 14 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Statistical Self-Similar Phenomenon in Language
  4. Characterizing LLMs via Self-Similarity
  5. Recurrence Structure and Correlation Dimension of Language
  6. Correlation Dimension Applied to Language.
  7. Textual Skips as Recurrences.
  8. Sufficiency of Next-Token Log-probabilities
  9. Empirical Convergence of Correlation Dimension
  10. Characterizing Language Models Using Correlation Dimension
  11. Correlation Dimension and Textual Complexity
  12. Effect of Contextual Constraints on Correlation Dimension
  13. Three-Stage Evolution in LLM Pre-Training
  14. Hallucination vs. Memorization: Diverging Correlation Dimension on Knowledge-Intensive Texts
  15. Correlation Dimension for Quantifying Degeneration in Text Generation
  16. ...and 29 more sections

Key Result

Theorem 2

Consider a fractal set embedded in $\mathbb{R}^D$ with Hausdorff dimension (or box-counting dimension) $d$ ($\le D$). Given a random linear projection from $\mathbb{R}^D$ to $\mathbb{R}^m$ ($m<D$), then with probability 1, the projected set's Hausdorff dimension $\tilde{d}$ satisfies:

Figures (8)

  • Figure 1: Experimental results on the "Newton's Philosophy" article of the Stanford Encyclopedia of Philosophy. (a-b) Examples of textual skips as predicted by the Pythia-12B model, illustrating both local and long-range recurrences in the text. (c) Segment of the recurrence plot for the log-probability vectors generated by the model; black dots indicate pairs of points within a specified distance threshold. (d) Correlation integral curves for six pre-trained language models (solid lines), compared to i.i.d. Gaussian noise in $\mathbb{R}^{50000}$ (dashed lines).
  • Figure 2: Left: Mean correlation dimension of the SEP dataset measured with various pre-trained LLMs. Right: A conjectured spectrum of correlation dimension for different types of texts / language models.
  • Figure 3: (a) Illustration of the Lin-Tegmark grammar generation process governed by the transition matrix $G$, parameterized by $q$. (b) Correlation integral curves for texts generated with different $q$ values, measured using the OpenLLaMA-13B model.
  • Figure 4: (a) Correlation integral curves on an article from the SEP dataset, measured by Pythia-1B model at different context lengths. (b) Mean correlation dimension on the SEP dataset (vertical axis), measured using three models at different context lengths, with respect to the negative perplexity (horizontal axis).
  • Figure 5: (a) Evolution of the mean correlation dimension for various language models on the SEP dataset during pre-training. (b) Notable case of two smaller models (Pythia-14M and 160M) that show a marked increase in correlation dimension (top) near the end of training, coinciding with a significant drop in in-context learning accuracy (bottom).
  • ...and 3 more figures

Theorems & Definitions (2)

  • Definition 1: Correlation dimension grassberger1983characterization
  • Theorem 2: Fractal Projection Theorems marstrand1954somefalconer2004fractal