Directed Metric Structures arising in Large Language Models
Stéphane Gaubert, Yiannis Vlassopoulos
TL;DR
The paper shows that conditional word-extension probabilities from large language models induce a directed metric on texts, turning negative log-probabilities into a rich geometric structure. It constructs directed metric polyhedra $P(\mathcal{L})$ and $Q(\mathcal{L})$ with Yoneda and co-Yoneda embeddings placing texts on extremal rays, and demonstrates that $P(\mathcal{L})$ is the $(\min,+)$-span of these rays while its dual $\widehat{P}(\mathcal{L})$ is the $(\min,+)$-span of the co-embeddings, linking semantics to geometry. A duality web connects text extensions and restrictions via extended semirings, Isbell completion, and lattice completions, with extremal rays corresponding to connected lower sets and offering a canonical decomposition of semantic space into principal and non-principal components. The framework provides a tropical-algebraic view of text representations, showing how short (one-word) extensions can be aggregated into longer text vectors through Boltzmann-weighted sums and how adding data preserves isometric embeddings, echoing attention mechanisms in transformers. Overall, the work delivers a mathematically explicit, category-informed picture of how language structure and meaning emerge from probabilistic extensions, with potential implications for representation learning and interpretability in LLMs.
Abstract
Large Language Models are transformer neural networks which are trained to produce a probability distribution on the possible next words to given texts in a corpus, in such a way that the most likely word predicted is the actual word in the training text. In this paper we find what is the mathematical structure defined by such conditional probability distributions of text extensions. Changing the view point from probabilities to -log probabilities we observe that the subtext order is completely encoded in a metric structure defined on the space of texts $\mathcal{L}$, by -log probabilities. We then construct a metric polyhedron $P(\mathcal{L})$ and an isometric embedding (called Yoneda embedding) of $\mathcal{L}$ into $P(\mathcal{L})$ such that texts map to generators of certain special extremal rays. We explain that $P(\mathcal{L})$ is a $(\min,+)$ (tropical) linear span of these extremal ray generators. The generators also satisfy a system of $(\min+)$ linear equations. We then show that $P(\mathcal{L})$ is compatible with adding more text and from this we derive an approximation of a text vector as a Boltzmann weighted linear combination of the vectors for words in that text. We then prove a duality theorem showing that texts extensions and text restrictions give isometric polyhedra (even though they look a priory very different). Moreover we prove that $P(\mathcal{L})$ is the lattice closure of (a version of) the so called, Isbell completion of $\mathcal{L}$ which turns out to be the $(\max,+)$ span of the text extremal ray generators. All constructions have interpretations in category theory but we don't use category theory explicitly. The categorical interpretations are briefly explained in an appendix. In the final appendix we describe how the syntax to semantics problem could fit in a general well known mathematical duality.