Hidden Holes: topological aspects of language models

TL;DR

The paper investigates the topology of representation manifolds in autoregressive language models by introducing perforation, a topology-based scalar φ(s) = ∑_{i=1}^{n} H_i log p_i, derived from Betti numbers and prime weights, to quantify hidden-space complexity. Using persistent homology tools (Vietoris-Rips filtrations, sliding-window embeddings) and simplicial mapping across LSTM and transformer architectures trained on natural and synthetic corpora, it tracks how topological features evolve with training. A key finding is that LSTM hidden states develop significant topological holes as perplexity decreases, while input embeddings lose topology, and transformer hidden layers exhibit much simpler topology, with synthetic data failing to reproduce the natural-language perforation pattern. The work suggests a link between computation graph topology and representation geometry, offering avenues for topology-informed regularization, architecture design, and natural-language detection via perforation.

Abstract

We explore the topology of representation manifolds arising in autoregressive neural language models trained on raw text data. In order to study their properties, we introduce tools from computational algebraic topology, which we use as a basis for a measure of topological complexity, that we call perforation. Using this measure, we study the evolution of topological structure in GPT based large language models across depth and time during training. We then compare these to gated recurrent models, and show that the latter exhibit more topological complexity, with a distinct pattern of changes common to all natural languages but absent from synthetically generated data. The paper presents a detailed analysis of the representation manifolds derived by these models based on studying the shapes of vector clouds induced by them as they are conditioned on sentences from corpora of natural language text. The methods developed in this paper are novel in the field and based on mathematical apparatus that might be unfamiliar to the target audience. To help with that we introduce the minimum necessary theory, and provide additional visualizations in the appendices. The main contribution of the paper is a striking observation about the topological structure of the transformer as compared to LSTM based neural architectures. It suggests that further research into mathematical properties of these neural networks is necessary to understand the operation of large transformer language models. We hope this work inspires further explorations in this direction within the NLP community.

Figures (26)

• Figure 1: An art installation by "Red" Hong Yi franco2016amazing showing the projection of a 3D sculpture onto a 2D plane. The projection can be misleading, while the original higher dimensional data has a more complex structure. In this analogy the text output from an LLM is the projected image. Our study develops tools to describe the shape of the internal representation manifolds (the sculpture) directly. We also track the evolution of those shapes throughout model training, and how they change with respect to choices of training data used.
• Figure 2: The torus (top) has a single connected component which corresponds to a single $0$-dimensional "hole". It has two $1$-dimensional holes - one measured by the class of loops wrapping around the main circle (represented by $a$) and another by the class of loops going through the middle hole (represented by $b$). These loops are independent because there is no way to continuously deform any of the loops in the first class into any of the loops in the second class. Finally, the torus has a single $2$-dimensional hole, which is generated by the tire shaped cavity inside its surface. These numbers of independent holes are the Betti numbers of the torus. Thus, the Betti numbers for the torus are [1, 2, 1]. The sphere (bottom) by contrast has no non-trivial loops in the first dimension. This is because every loop on a sphere can be continuously deformed into any other loop on its surface, and they are all contractible to a point. The Betti numbers of the sphere are thus [1, 0, 1]. Both of these are 2-dimensional manifolds embedded in a 3-dimensional ambient space, so the sequence of Betti numbers has only 3 elements. In general, for higher dimensional manifolds such as those we study in this work, the sequence will continue and the n-th number can be interpreted as counting n-dimensional cells wrapped around (n+1)-dimensional cavities in the manifold's topological structure.
• Figure 3: The main data analysis pipeline. Raw text is transformed into a sequence of high dimensional point clouds by the neural layers of a language model. The shapes of these point clouds are then analyzed from a topological perspective by computing persistent homology modules. The free ranks of these algebraic objects keep track of the number of holes emerging within the manifolds from which these point clouds are sampled. We also used additional topological analysis methods, which are explained in the appendices.
• Figure 4: A filtration of Vietoris-Rips complexes with distance parameter $\epsilon$ on a set of points embedded in an ambient metric space, and the associated persistent homology barcodes. ghrist2008barcodes
• Figure 5: Inducing topological structure from a point cloud representing noisy samples from a neighborhood of a 1-dimensional submanifold ($\mathbb{S}^2$) of a 2-dimensional ambient embedding space ($\mathbb{R}^2$). This method can be adjusted to produce simplicial complex summaries in all dimensions singh2007topological.