When Models Manipulate Manifolds: The Geometry of a Counting Task

TL;DR

This work probes how a large language model performs linebreaking in fixed-width text by revealing that character counts are encoded on a $1$-dimensional manifold embedded in a $6$-dimensional subspace, with curvature arising from distributed attention-head contributions. The boundary to the line end is detected by boundary Heads that twist the count and width representations, and the newline decision emerges from near-orthogonal stacking of the remaining-characters and next-token-length counts, enabling a linear separator with high accuracy. Causal interventions and visual illusion experiments demonstrate the stability and fragility of the counting mechanism, underscoring the distributed, geometry-enabled computations in early layers. Overall, the study highlights the value of combining feature-based and geometric interpretability to reveal how neural networks perform perceptual tasks and suggests avenues for automatic discovery of geometric structures in model internals.

Abstract

Language models can perceive visual properties of text despite receiving only sequences of tokens-we mechanistically investigate how Claude 3.5 Haiku accomplishes one such task: linebreaking in fixed-width text. We find that character counts are represented on low-dimensional curved manifolds discretized by sparse feature families, analogous to biological place cells. Accurate predictions emerge from a sequence of geometric transformations: token lengths are accumulated into character count manifolds, attention heads twist these manifolds to estimate distance to the line boundary, and the decision to break the line is enabled by arranging estimates orthogonally to create a linear decision boundary. We validate our findings through causal interventions and discover visual illusions--character sequences that hijack the counting mechanism. Our work demonstrates the rich sensory processing of early layers, the intricacy of attention algorithms, and the importance of combining feature-based and geometric views of interpretability.

Figures (63)

• Figure 1: Left: a description of the linebreaking task. Center: example feature families we study. Right: a highlight of the geometry of representations and computations.
• Figure 2: A pair of prompts highlighting the difficulty of the linebreaking task.
• Figure 3: Attribution Graph for Claude 3.5 Haiku’s prediction of a newline in the aluminum prompt. We see features relating to “width of the previous line” and “position in the current line” which together activate features for “distance from line limit”. Combined with features for the planned next word, these features activate “predict newline” features.
• Figure 4: Key steps in the linebreaking behavior can be described in terms of the construction and manipulation of manifolds.
• Figure 5: A family of features representing the current character count in a line of text. The tuning curve of the features’ activity increases at larger line character counts.