Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Symmetry in language statistics shapes the geometry of model representations

Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri

TL;DR

This work proposes that translation symmetry in language co-occurrence statistics fundamentally shapes neural representations. By linking the co-occurrence matrix ${\bm{M}}^ op$ to embedding geometry, the authors derive Fourier-like embedding structures for periodic and open semantic continua, predicting Lissajous projections and robust linear decodability from sparse coordinates. They demonstrate these predictions across word embeddings, text embeddings, and large language models, and further show that a collective latent-variable mechanism yields circulant PMI structures robust to perturbations. The results provide a unifying, symmetry-based explanation for spatial, temporal, and circular manifolds in language representations, with implications for understanding model cognition and guiding robust representation learning.

Abstract

Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM representations: for example, calendar months organize into a circle, years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded by a linear probe. We show that the statistics of language exhibit a translation symmetry -- e.g., the co-occurrence probability of two months depends only on the time interval between them -- and we prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models. Moreover, we find that these structures persist even when the co-occurrence statistics are strongly perturbed (for example, by removing all sentences in which two months appear together) and at moderate embedding dimension. We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable. We empirically validate this theoretical framework in word embedding models, text embedding models, and large language models.

Symmetry in language statistics shapes the geometry of model representations

TL;DR

This work proposes that translation symmetry in language co-occurrence statistics fundamentally shapes neural representations. By linking the co-occurrence matrix to embedding geometry, the authors derive Fourier-like embedding structures for periodic and open semantic continua, predicting Lissajous projections and robust linear decodability from sparse coordinates. They demonstrate these predictions across word embeddings, text embeddings, and large language models, and further show that a collective latent-variable mechanism yields circulant PMI structures robust to perturbations. The results provide a unifying, symmetry-based explanation for spatial, temporal, and circular manifolds in language representations, with implications for understanding model cognition and guiding robust representation learning.

Abstract

Although learned representations underlie neural networks' success, their fundamental properties remain poorly understood. A striking example is the emergence of simple geometric structures in LLM representations: for example, calendar months organize into a circle, years form a smooth one-dimensional manifold, and cities' latitudes and longitudes can be decoded by a linear probe. We show that the statistics of language exhibit a translation symmetry -- e.g., the co-occurrence probability of two months depends only on the time interval between them -- and we prove that the latter governs the aforementioned geometric structures in high-dimensional word embedding models. Moreover, we find that these structures persist even when the co-occurrence statistics are strongly perturbed (for example, by removing all sentences in which two months appear together) and at moderate embedding dimension. We show that this robustness naturally emerges if the co-occurrence statistics are collectively controlled by an underlying continuous latent variable. We empirically validate this theoretical framework in word embedding models, text embedding models, and large language models.
Paper Structure (58 sections, 5 theorems, 112 equations, 16 figures)

This paper contains 58 sections, 5 theorems, 112 equations, 16 figures.

Table of Contents

  1. Introduction
  2. Preliminaries and Related Work
  3. Predicting word embedding geometry
  4. Related work
  5. Co-occurrence model with symmetry
  6. Predicting embedding geometry for periodic lattices
  7. Predicting embedding geometry for $D=1$ lattices
  8. New predictions of the co-occurrence model
  9. Beyond the lattice approximation
  10. Collective effects control the embedding of space and time
  11. Co-occurrence arising from a shared continuous latent variable
  12. Diagonalization of the PMI
  13. Robustness of embedding to removal of the PMI elements:
  14. Joint continuous and binary model
  15. Discussion
  16. ...and 43 more sections

Key Result

Proposition 0

Let $\mathcal{S}$ correspond to a periodic semantic lattice. Assume asmp:translationsymmetry holds on $\mathcal{S}$ with kernel $C(\cdot)$. Assume the embedding dimension $d\geq \rank({\bm{M}}^\star)$. Define the set of allowed wavevectors $\{{\bm{k}}_\mu\}_{\mu=1}^{|\mathcal{S}|}$, each satisfying

Figures (16)

  • Figure 1: Cyclic representation manifolds arise from translation symmetry in co-occurrence statistics. We visualize the top principal components of the word representations for calendar time and historical time, and we show the representation vectors' Gram matrix. In the top row, we predict both embedding geometry and their Gram matrix using the parametric curves analytically derived in \ref{['cor:exp-periodic']} and \ref{['prop:exp-open']}. We empirically compare these predictions to both word embeddings trained on Wikipedia (middle row) and to LLM internal representations (bottom row). The excellent agreement provides evidence that translation symmetry in co-occurrence statistics drives the formation of representational manifolds in neural networks. See \ref{['app:experiments']} for experimental details.
  • Figure 2: Lattice translation symmetry implies Lissajous curves in embedding space and linear coordinate decoding. We empirically validate two novel predictions of our theory. (Left.) When projected on any two principal components, word embeddings (e.g., for historical years) display Lissajous curves whose amplitudes, phases, and frequencies are analytically predicted by \ref{['prop:exp-open']}. (Right.) Linear probes can decode the numerical year from the sinusoidal embedding modes. Each included higher-frequency mode increases the temporal resolution; the error decay rate is predicted by \ref{['prop:decode']}. At the interpolation threshold $r=n_\text{train}$, training error quickly vanishes and the test error exhibits the expected double descent peak.
  • Figure 3: Translation symmetry in geographic data yields top embedding modes with slow spatial variation. We depict the top PCA modes (i.e., top eigenfunctions of the representational Gram matrix) for the 48 contiguous US states. In the leftmost column, we show the modes obtained from a theoretical model in which the co-occurrence probability of two states depends only on the Euclidean distance between their centroids, in geographic coordinates. Since the states do not lie on a perfect lattice, the theoretical eigenmodes are not exactly 2D plane waves; however, they still qualitatively exhibit slowly-varying oscillations. We compare to word embeddings, text embeddings, and LLM internal representations, and find good agreement. We observe that low-dimensional word embeddings more faithfully reflect the geographic semantic attributes (see \ref{['sec:collective']} for an explanation as to why low-dimensional embeddings can be more robust to noise). See \ref{['app:experiments']} for experimental details.
  • Figure 4: Circular embedding of the months reconstructed without the month-month co-occurrences.Left: we consider the English Wikipedia ${\bm{M}}^*$ matrix ($V=2.5\times10^4$), with the month-month co-occurrences ablated. Embeddings of dimension $d = 10^3\ll V$ nonetheless exhibit a circular geometry (for a PCA over the months, excluding May) with the correct ordering, and lead to a Gram matrix that closely approximates the original month-month co-occurrences. That this is possible implies the existence of redundant time-of-the-year latent variables affecting the rest of the vocabulary. Right: we identify a subset of highly seasonal words and their phases (see \ref{['sec:appdx_reconstruction_data']} for details), and show that just a handful of such seasonal words are sufficient to reconstruct the month ordering. See \ref{['app:experiments']} for experimental details.
  • Figure 5: Empirical spectral distribution of ${\bm{M}}^\star$ and PMI. We show the histogram of eigenvalues of both ${\bm{M}}^\star$ (left two plots) and the PMI matrix (right two plots). For each, we plot both the logarithmic density $p(t)$ for $t\coloneqq\log(|\lambda|)$ as well as $p(\lambda)$ directly. The latter shows that the spectrum is roughly symmetric, with the PSD and NSD components being of comparable rank ($\rank {\bm{M}}^+ \approx \rank {\bm{M}}^-$ as defined in \ref{['eq:qwem-psd-nsd-decomp']}). The former displays a linear growth, indicating a spectral bulk with no divergence at zero, followed by a steep drop-off, which may be interpreted as a bulk spectral edge. Many eigenvalues exist beyond the spectral edge, suggesting that these are semantic signals.
  • ...and 11 more figures

Theorems & Definitions (5)

  • Proposition 0: Fourier embedding geometry (simplified)
  • Corollary 0: Periodized exponential kernel (simplified)
  • Proposition 0: Exponential kernel, open BC (simplified)
  • Proposition 0: Linear coordinate decoding (simplified)
  • Theorem 1: Spectrum of the combined seasonal--attribute PMI