Bridging the Dimensional Chasm: Uncover Layer-wise Dimensional Reduction in Transformers through Token Correlation

TL;DR

This work introduces a geometric, token-level framework to resolve how high-dimensional transformer representations consolidate into low-dimensional semantic structure. By defining a layer-wise correlator E(ξ) and modeling Transformer layers as dimensional projectors, it uncovers an expansion–contraction dynamic: tokens first diffuse into a working space E* of dimension d_model, then contract into a semantic space E_machine of dimension d_machine. A key finding is that smaller d_model correlates with better task performance, while larger d_machine grows with model size, suggesting a trade-off between high-dimensional feature capacity and efficient semantic compression. Practically, the correlator-based metrics offer rapid, task-independent diagnostics to compare models and potentially guide training objectives without extensive task-specific evaluations.

Abstract

The geometric evolution of token representations in large language models (LLMs) presents a fundamental paradox: while human language inherently organizes semantic information in low-dimensional spaces ($\sim 10^1$ dimensions), modern LLMs employ high-dimensional embeddings ($\sim 10^3$ dimensions) processed through Transformer architectures. To resolve this paradox, this work bridges this conceptual gap by developing a geometric framework that tracks token dynamics across Transformers layers. Through layer-wise analysis of intrinsic dimensions across multiple architectures, we reveal an expansion-contraction pattern where tokens diffuse to a "working space" and then progressively project onto lower-dimensional submanifolds. Our finding implies a negative correlation between the working space dimension and parameter-sensitive performance of the LLMs, and indicates that effective models tend to compress tokens into approximately 10-dimensional submanifolds, closely resembling human semantic spaces. This work not only advances LLM interpretability by reframing Transformers layers as projectors that mediate between high-dimensional computation and low-dimensional semantics, but also provides practical tools for model diagnostics that do not rely on task-specific evaluations.

Figures (9)

• Figure 1: Spectrum for sentence sample 1 in Appendix \ref{['subsec:1']}. We clip eigenvalues whose values are less than $10^{-8}$ and set them to $10^{-8}$.
• Figure 2: How the correlator changes in each layer. The trends of the correlator and Cosine Similarity are consistent.
• Figure 3: Schematic Diagram of Projection. This figure illustrates how the projection of word vectors affects the angles between them.
• Figure 4: How correlator changes in layers of a random model(correlator axis is log scaled)
• Figure 5: Evolution for correlator calculated by other models