Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Latent Semantic Manifolds in Large Language Models

Mohamed A. Mabrok

Abstract

Large Language Models (LLMs) perform internal computations in continuous vector spaces yet produce discrete tokens -- a fundamental mismatch whose geometric consequences remain poorly understood. We develop a mathematical framework that interprets LLM hidden states as points on a latent semantic manifold: a Riemannian submanifold equipped with the Fisher information metric, where tokens correspond to Voronoi regions partitioning the manifold. We define the expressibility gap, a geometric measure of the semantic distortion from vocabulary discretization, and prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap via the coarea formula. We validate these predictions across six transformer architectures (124M-1.5B parameters), confirming universal hourglass intrinsic dimension profiles, smooth curvature structure, and linear gap scaling with slopes 0.87-1.12 (R^2 > 0.985). The margin distribution across models reveals a persistent hard core of boundary-proximal representations invariant to scale, providing a geometric decomposition of perplexity. We discuss implications for architecture design, model compression, decoding strategies, and scaling laws

Latent Semantic Manifolds in Large Language Models

Abstract

Large Language Models (LLMs) perform internal computations in continuous vector spaces yet produce discrete tokens -- a fundamental mismatch whose geometric consequences remain poorly understood. We develop a mathematical framework that interprets LLM hidden states as points on a latent semantic manifold: a Riemannian submanifold equipped with the Fisher information metric, where tokens correspond to Voronoi regions partitioning the manifold. We define the expressibility gap, a geometric measure of the semantic distortion from vocabulary discretization, and prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap via the coarea formula. We validate these predictions across six transformer architectures (124M-1.5B parameters), confirming universal hourglass intrinsic dimension profiles, smooth curvature structure, and linear gap scaling with slopes 0.87-1.12 (R^2 > 0.985). The margin distribution across models reveals a persistent hard core of boundary-proximal representations invariant to scale, providing a geometric decomposition of perplexity. We discuss implications for architecture design, model compression, decoding strategies, and scaling laws
Paper Structure (67 sections, 15 theorems, 72 equations, 11 figures, 4 tables)

This paper contains 67 sections, 15 theorems, 72 equations, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Token Space and Embedding
  5. Contextual Representations
  6. Token Distribution
  7. The Latent Semantic Manifold
  8. The Manifold Hypothesis for LLM Representations
  9. Tangent Space and Local Structure
  10. Local Coordinate Charts
  11. Riemannian Structure and the Fisher Information Metric
  12. The Induced Metric
  13. The Fisher Information Metric
  14. The Restricted Fisher Metric
  15. Token Generation as Voronoi Projection
  16. ...and 52 more sections

Key Result

Proposition 5.2

Under the softmax parameterization with logits $\ell_t(h) = \langle w_t, h \rangle + b_t$, the Fisher information matrix in ambient coordinates is where $p = p(\cdot \mid h) \in \mathbb{R}^N$ is the token probability vector and $\Sigma_p = \mathop{\mathrm{diag}}\nolimits(p) - p \, p^\top$ is the covariance matrix of the categorical distribution.

Figures (11)

  • Figure 1: Voronoi tessellation of the semantic manifold $\mathcal{M}$. Each token embedding $e_t$ generates a Voronoi region $R_t = \{h \in \mathcal{M} : \|h - e_t\| \le \|h - e_{t'}\| \;\forall\, t'\}$ (colored cells). The Voronoi boundary $\partial\mathcal{V}$ (solid gray lines) is the locus of maximal ambiguity where two or more tokens tie. For a hidden state $h$ near the boundary between $R_{t_1}$ and $R_{t_3}$, the Voronoi margin$m(h) = \ell_{t^*}(h) - \ell_{t^{**}}(h)$ is small, placing $h$ inside the expressibility gap$\mathcal{G}_\varepsilon = \{h : m(h) < \varepsilon\}$ (orange shaded strip). Points deep inside a region have large margins and are expressed unambiguously; points in $\mathcal{G}_\varepsilon$ represent semantic states that the finite vocabulary cannot capture with high confidence.
  • Figure 2: Intrinsic dimension profile for GPT-2 (124M). (a) TWO-NN and MLE estimates by layer, with the ambient dimension $d=768$ indicated as annotation (note the $\approx$35$\times$ gap between peak intrinsic dimension and ambient dimension). (b) Dimension utilization as percentage of $d$. (c) Layer-to-layer dimension change $\Delta\hat{k}$, revealing the expansion--contraction transition.
  • Figure 3: Cross-architecture comparison of intrinsic dimension for small-scale models (124M--160M parameters). (a) TWO-NN dimension versus normalized depth. (b) MLE dimension versus normalized depth. (c) Dimension utilization as percentage of ambient dimension. (d) Summary statistics. All three architectures exhibit the characteristic hourglass pattern despite differing training data and hyperparameters.
  • Figure 4: Cross-architecture comparison of intrinsic dimension for large-scale models (1B--1.5B parameters). The hourglass pattern is preserved and becomes more pronounced: peak dimensions are similar ($\hat{k} \approx 20$--$22$) despite the ambient dimension doubling or tripling, resulting in lower utilization ratios.
  • Figure 5: Detailed curvature analysis for GPT-2 XL. (a) Distribution of local PCA curvature values with median line. (b) Curvature versus prediction entropy. (c) Second fundamental form $\|\mathrm{I\!I}\|$ distribution. (d) Layer-wise curvature profile on logarithmic scale with interquartile range bands.
  • ...and 6 more figures

Theorems & Definitions (45)

  • Definition 3.1: Contextual representation set
  • Remark 4.2
  • Definition 4.3: Semantic tangent vector
  • Definition 4.4: Semantic coordinate chart
  • Remark 4.5: Interpretation of semantic coordinates
  • Definition 5.1: Fisher information metric on $\mathcal{M}$
  • Proposition 5.2: Explicit form of the Fisher metric
  • proof
  • Remark 5.3
  • Definition 5.4: Restricted Fisher metric
  • ...and 35 more