The Representational Geometry of Number

TL;DR

The paper investigates whether conceptual representations share a common geometric framework or decompose into task-specific subspaces. Using numbers as a testbed and large language models as high‑dimensional substrates, it reveals a stable, shared relational structure across tasks that is preserved through linear transformations, while task-specific representations occupy largely separable subspaces. Low Procrustes disparity ($M^2 \approx 0.010$) and high SVCCA compatibility ($\approx 0.80$–$0.90$) indicate that, despite subspace separation, the underlying relations among numbers are coherent and transferable. These findings provide a mechanistic, geometry‑driven account of how representations balance shared structure with task flexibility, with implications for conceptual spaces theory and cognitive science, including insights into why certain numerical abilities may falter in dyscalculia.

Abstract

A central question in cognitive science is whether conceptual representations converge onto a shared manifold to support generalization, or diverge into orthogonal subspaces to minimize task interference. While prior work has discovered evidence for both, a mechanistic account of how these properties coexist and transform across tasks remains elusive. We propose that representational sharing lies not in the concepts themselves, but in the geometric relations between them. Using number concepts as a testbed and language models as high-dimensional computational substrates, we show that number representations preserve a stable relational structure across tasks. Task-specific representations are embedded in distinct subspaces, with low-level features like magnitude and parity encoded along separable linear directions. Crucially, we find that these subspaces are largely transformable into one another via linear mappings, indicating that representations share relational structure despite being located in distinct subspaces. Together, these results provide a mechanistic lens of how language models balance the shared structure of number representation with functional flexibility. It suggests that understanding arises when task-specific transformations are applied to a shared underlying relational structure of conceptual representations.

Figures (6)

• Figure 1: t-SNE projections of number representations extracted from BERT (left) and Qwen2.5-Math (right) across numerical tasks. Each point corresponds to a representation of a number (1–9). Colors indicate task categories, while marker shapes represent formats (digit vs. English word).
• Figure 2: Three effects fitted across models. Cosine similarity between number embeddings (Digit) is plotted against numerical distance (left), size (middle), and ratio (right) for each task. Top row: BERT; bottom row: Qwen2.5-Math. Each line represents a linear (left/middle) or logarithmic (right) fit for a given task (color-coded), with absolute Pearson’s r and significance level indicated in legend ($p<0.05:$ *, $p<0.01:$ **, $p<0.001:$ ***).
• Figure 3: Parity: Odd vs. Even
• Figure 4: Primality: Prime vs. Composite
• Figure 6: Task-specific representation (Digit) alignment in Qwen2.5-Math. The left panel shows an asymmetric Subspace Overlap Ratio, while the right panel shows symmetric SVCCA similarity.