Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations

Yize Zhao, Christos Thrampoulidis

TL;DR

The paper investigates how next-token prediction optimization intrinsically organizes linguistic semantics in word and context representations, revealing that learned geometry aligns with the singular value decomposition of a centered data-sparsity matrix.By introducing an unconstrained features model (NTP-UFM) and an orthant-based clustering approach, it shows that latent semantic concepts are encoded in the signs and magnitudes of SVD factors and can be extracted as interpretable categories through sign-pattern combinations.The authors demonstrate both theoretically and empirically that concepts associated with larger singular values are learned earlier, yielding a coarse-to-fine semantic emergence, and validate these insights on synthetic data and multiple pretrained models across languages and architectures.This work connects classical distributional semantics with neural-collapse geometry, providing a principled optimization-based explanation for semantic organization and offering a practical method for discovering interpretable semantics from language-model representations.

Abstract

We investigate how next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text. Our analysis, based on a tractable mathematical model and controlled synthetic data, reveals that NTP implicitly guides models to factor a centered support matrix encoding context-to-next-token co-occurrence patterns via singular value decomposition (SVD). While models never explicitly construct this matrix, learned word and context embeddings converge to its SVD factors, with singular vectors encoding latent semantic concepts through their sign patterns. We demonstrate that concepts corresponding to larger singular values are learned earlier during training, yielding a natural semantic hierarchy where broad categories emerge before fine-grained ones. This insight motivates orthant-based clustering, a method that combines concept signs to identify interpretable semantic categories. We validate our findings on synthetic datasets and pretrained language models, recovering diverse semantic structures such as grammatical categories, named entity types, and topical distinctions (medical, entertainment). Our work bridges classical distributional semantics and neural collapse geometry, characterizing how gradient-based optimization implicitly determines both the matrix representation and factorization method that encode semantic structure.

Geometry of Semantics in Next-Token Prediction: How Optimization Implicitly Organizes Linguistic Representations

TL;DR

The paper investigates how next-token prediction optimization intrinsically organizes linguistic semantics in word and context representations, revealing that learned geometry aligns with the singular value decomposition of a centered data-sparsity matrix.By introducing an unconstrained features model (NTP-UFM) and an orthant-based clustering approach, it shows that latent semantic concepts are encoded in the signs and magnitudes of SVD factors and can be extracted as interpretable categories through sign-pattern combinations.The authors demonstrate both theoretically and empirically that concepts associated with larger singular values are learned earlier, yielding a coarse-to-fine semantic emergence, and validate these insights on synthetic data and multiple pretrained models across languages and architectures.This work connects classical distributional semantics with neural-collapse geometry, providing a principled optimization-based explanation for semantic organization and offering a practical method for discovering interpretable semantics from language-model representations.

Abstract

We investigate how next-token prediction (NTP) optimization leads language models to extract and organize semantic structure from text. Our analysis, based on a tractable mathematical model and controlled synthetic data, reveals that NTP implicitly guides models to factor a centered support matrix encoding context-to-next-token co-occurrence patterns via singular value decomposition (SVD). While models never explicitly construct this matrix, learned word and context embeddings converge to its SVD factors, with singular vectors encoding latent semantic concepts through their sign patterns. We demonstrate that concepts corresponding to larger singular values are learned earlier during training, yielding a natural semantic hierarchy where broad categories emerge before fine-grained ones. This insight motivates orthant-based clustering, a method that combines concept signs to identify interpretable semantic categories. We validate our findings on synthetic datasets and pretrained language models, recovering diverse semantic structures such as grammatical categories, named entity types, and topical distinctions (medical, entertainment). Our work bridges classical distributional semantics and neural collapse geometry, characterizing how gradient-based optimization implicitly determines both the matrix representation and factorization method that encode semantic structure.
Paper Structure (27 sections, 2 theorems, 18 equations, 15 figures, 1 table)

This paper contains 27 sections, 2 theorems, 18 equations, 15 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background: Geometry of Word and Context Representations
  3. Geometry of Semantics
  4. Identifying Concepts as Latent Signals
  5. Interpretable Semantics via Orthant-based Clustering
  6. Semantic Emergence
  7. Experimental Validation
  8. Methodology
  9. Results
  10. Related Work
  11. Conclusion
  12. Limitations
  13. Acknowledgments
  14. Paper Organization
  15. Extended Discussion on Related Work
  16. ...and 12 more sections

Key Result

Theorem 1

Consider gradient-descent minimization of NTP-UFM (Eq. eq:ufm) with square loss. Assume infinitesimal step-size and spectral initialization ${\bm{W}}(0)=e^{-\delta}\bm{U}\mathbf{R}^\top, {\bm{H}}(0)=e^{-\delta}\mathbf{R}{\bm{V}}^\top$ for partial orthogonal matrix $\mathbf{R}\in\mathbb{R}^{d\times r Therefore, the $i$-th concept converges ($a_i(t)\rightarrow 1$) with exponential rate of $2\sigma_i

Figures (15)

  • Figure 1: Illustration of C1-3. Details in Sec. \ref{['sec:concept geometry']}. (A) SVD of the data-sparsity matrix $\widetilde{\bm{S}}$ (Eq. \ref{['eq:smatbar']}), derived from the training data in Fig. \ref{['fig:motivation']}(C). Columns of $\bm{U}$ and ${\bm{V}}$, serve as word and context concept basis vectors, respectively: Their entries encode an association score with sign indicating the direction (association/opposition) between word/contexts and concepts. The singular values indicate the strength of each concept. (B) Combining a few top concepts according to the specific sign patterns of concept basis, which we call orthant-based clustering, reveals human-interpretable semantics. (C) Gradient-descent-based NTP optimization leads to concepts corresponding to larger singular values being learned first. This corresponds to the emergence of coarse semantics before fine-grained ones, which we empirically validate with visualizations of the trained model's output confusion matrices.
  • Figure 2: Optimizing NTP on (context,next-word) input pairs yields a specific geometry of representation of those inputs; orthant-based clustering reveals how this geometry is organized on the basis of interpretable semantic categories.(A) Empirical 3D PCA visualization of learned word(dots)/context(triangular marks) embeddings from two-layer transformer trained on the dataset in (C) reveals a clear clustering by semantic category (plants in green/blue, animals in red/orange), despite no explicit semantic signal at training. (B) We show (Sec. \ref{['sec:orthant']}) that semantic categories are organized into a hierarchy of sub-orthants defined by concepts combined with respect to their sign patters. For example, Felines occupy a 2D sub-orthant (combining the second and fourth concepts), while Mammals and Animals occupy progressively larger sub-orthants. (C) The sparse training data matrix (rows = attributes, columns = subjects) follows the template: “The organism that [attribute] is [subject].”
  • Figure 3: Semantics from orthant-based clustering on $\widetilde{\bm{S}}$ in Simplified TinyStories (Top) and Simplified Wikitext (Bottom). Semantic interpretations and combination configurations are shown in the title.
  • Figure 4: Semantics identified by orthant-based clustering on GPT-2’s word embeddings.
  • Figure 5: Semantics from transformers of different depth, both trained on Simplified TinyStories.
  • ...and 10 more figures

Theorems & Definitions (4)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • proof