Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Token embeddings violate the manifold hypothesis

Michael Robinson, Sourya Dey, Tony Chiang

TL;DR

This work interrogates the assumption that LLM token embeddings lie on a low-dimensional manifold by introducing a fiber bundle hypothesis and two hypothesis tests. The authors develop a rigorous framework linking ball-volume growth to geometric structure, showing that token spaces often exhibit non-manifold or non-fiber-bundle behavior, and that singularities can persist under typical context windows. Through synthetic datasets and four open-source LLMs (GPT2, Llemma7B, Mistral7B, Pythia6.9B), they demonstrate systematic rejections of manifold structure and, in several cases, fiber-bundle structure, with larger-radius neighborhoods revealing stronger deviations. These findings imply inherent irregularities in token topology that can affect prompt stability and semantic interpretation, highlighting the limitations of assuming smooth token spaces in LLM analysis and prompting new lines of inquiry into tokenization and model behavior.

Abstract

A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space. If this space differs from our assumptions, our comprehension of and conclusions about the LLM will likely be flawed. We elucidate the structure of the token embeddings both empirically and theoretically. We present a novel statistical test assuming that the neighborhood around each token has a relatively flat and smooth structure as the null hypothesis. Failing to reject the null is uninformative, but rejecting it at a specific token $ψ$ implies an irregularity in the token subspace in a $ψ$-neighborhood, $B(ψ)$. The structure assumed in the null is a generalization of a manifold with boundary called a \emph{smooth fiber bundle} (which can be split into two spatial regimes -- small and large radius), so we denote our new hypothesis test as the ``fiber bundle hypothesis.'' By running our test over several open-source LLMs, each with unique token embeddings, we find that the null is frequently rejected, and so the evidence suggests that the token subspace is not a fiber bundle and hence also not a manifold. As a consequence of our findings, when an LLM is presented with two semantically equivalent prompts, if one prompt contains a token implicated by our test, the response to that prompt will likely exhibit less stability than the other.

Token embeddings violate the manifold hypothesis

TL;DR

This work interrogates the assumption that LLM token embeddings lie on a low-dimensional manifold by introducing a fiber bundle hypothesis and two hypothesis tests. The authors develop a rigorous framework linking ball-volume growth to geometric structure, showing that token spaces often exhibit non-manifold or non-fiber-bundle behavior, and that singularities can persist under typical context windows. Through synthetic datasets and four open-source LLMs (GPT2, Llemma7B, Mistral7B, Pythia6.9B), they demonstrate systematic rejections of manifold structure and, in several cases, fiber-bundle structure, with larger-radius neighborhoods revealing stronger deviations. These findings imply inherent irregularities in token topology that can affect prompt stability and semantic interpretation, highlighting the limitations of assuming smooth token spaces in LLM analysis and prompting new lines of inquiry into tokenization and model behavior.

Abstract

A full understanding of the behavior of a large language model (LLM) requires our grasp of its input token space. If this space differs from our assumptions, our comprehension of and conclusions about the LLM will likely be flawed. We elucidate the structure of the token embeddings both empirically and theoretically. We present a novel statistical test assuming that the neighborhood around each token has a relatively flat and smooth structure as the null hypothesis. Failing to reject the null is uninformative, but rejecting it at a specific token implies an irregularity in the token subspace in a -neighborhood, . The structure assumed in the null is a generalization of a manifold with boundary called a \emph{smooth fiber bundle} (which can be split into two spatial regimes -- small and large radius), so we denote our new hypothesis test as the ``fiber bundle hypothesis.'' By running our test over several open-source LLMs, each with unique token embeddings, we find that the null is frequently rejected, and so the evidence suggests that the token subspace is not a fiber bundle and hence also not a manifold. As a consequence of our findings, when an LLM is presented with two semantically equivalent prompts, if one prompt contains a token implicated by our test, the response to that prompt will likely exhibit less stability than the other.

Paper Structure

This paper contains 24 sections, 3 theorems, 10 equations, 8 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background and related work
  4. Mathematical background
  5. Theoretical insights
  6. Volume versus radius for fiber bundles
  7. Context does not persistently resolve singularities
  8. Methods
  9. Results
  10. Synthetic datasets
  11. LLM token embeddings
  12. Discussion
  13. Limitations
  14. Supplementary Material
  15. Additional discussion of the literature
  16. ...and 9 more sections

Key Result

Theorem 1

Suppose that $T$ is a compact, finite-dimensional Riemannian manifold with boundary, with a volume form $v$ satisfying $v(T) < \infty$, and let $p: T \to S$ be a fiber bundle. If $e: T \to \mathbb{R}^{\ell}$ is a smooth embedding with reach $\tau$, then there is a function $\rho : e(T) \to [0,\tau]$ where $B_r(\psi)$ is the ball of radius $r$ centered at $\psi$, and the asymptotic limits are valid

Figures (8)

  • Figure 1: The histogram of local dimensions estimated near tokens in GPT2 (robinson2024structuretokenspacelarge). This histogram shows a mixture with at least three distinct peaks.
  • Figure 2: Interpretation of the manifold and fiber bundle tests: the manifold test detects any slope changes in the log volume versus log radius curves, while the fiber bundle test only detects slope increases.
  • Figure 3: This figure shows the application of this paper's manifold hypothesis test on three synthetic datasets: (a) a manifold, the unit sphere in $\mathbb{R}^3$, (b) a fiber bundle, the cusp surface $z^8 + x^2 + y^2 = 0$ in $\mathbb{R}^3$, (c) neither, a "lollipop" space, and (d) a neighborhood of the token ember in Mistral7B. Points are colored by the $p$ values of the manifold hypothesis test; darker colors mean reject the manifold hypothesis.
  • Figure 4: Data flow in a typical LLM. A sequence of tokens forming the query is converted via the token input embedding $e^w$ into the initial context window, as a point in the latent space $X^w$. Each of these windows in the latent space are converted, token-by-token, into probability distributions via $f$ into the single token latent space $X$. From these, each token presented in the output (in the set $Y$) is obtained via a random draw. These output tokens are then used for subsequent windows.
  • Figure 5: Our method applied to a fiber bundle in $\mathbb{R}^2$. The vertical direction is the base space (long spatial scale), while the horizontal direction represents the fibers space (short spatial scale). Gray points on the right frame show estimates from a random sampling of points in the strip; the solid line shows the theoretical area versus radius curve.
  • ...and 3 more figures

Theorems & Definitions (9)

  • Theorem 1
  • proof
  • Theorem 2
  • proof
  • Definition 1
  • proof
  • Lemma 1
  • proof
  • proof