Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models

Samuel Cyrenius Anderson

TL;DR

This paper shows that scaling large language models reshapes reasoning through domain-specific geometric reorganizations rather than a uniform performance boost. By analyzing over 25,000 chain-of-thought trajectories across Law, Science, Code, and Math and two model scales, it uncovers three phases—Crystallization in Law, Liquidity in Science/Math, and Lattice in Code—each with distinct effects on global dimension $d_{95}$, intrinsic dimension $d_{\mathrm{mle}}$, alignment, and clustering, plus a universal oscillatory coherence of about $-0.4$. It introduces Neural Reasoning Operators to map initial to terminal hidden states and demonstrates that favorable geometry predicts operator learnability, enabling amortized inference in crystalline domains. The findings imply that the cost of thought depends on manifold geometry and point toward geometry-aware deployment, compression, and safety considerations, with practical methods for operator-based inference and a reproducible measurement suite. Overall, the work provides a geometry-centric map of how scale restructures reasoning, offering a blueprint for domain-specific acceleration and more reliable deployment of high-stakes AI systems.

Abstract

Scale does not uniformly improve reasoning - it restructures it. Analyzing 25,000+ chain-of-thought trajectories across four domains (Law, Science, Code, Math) and two scales (8B, 70B parameters), we discover that neural scaling laws trigger domain-specific phase transitions rather than uniform capability gains. Legal reasoning undergoes Crystallization: 45% collapse in representational dimensionality (d95: 501 -> 274), 31% increase in trajectory alignment, and 10x manifold untangling. Scientific and mathematical reasoning remain Liquid - geometrically invariant despite 9x parameter increase. Code reasoning forms a discrete Lattice of strategic modes (silhouette: 0.13 -> 0.42). This geometry predicts learnability. We introduce Neural Reasoning Operators - learned mappings from initial to terminal hidden states. In crystalline legal reasoning, our operator achieves 63.6% accuracy on held-out tasks via probe decoding, predicting reasoning endpoints without traversing intermediate states. We further identify a universal oscillatory signature (coherence ~ -0.4) invariant across domains and scales, suggesting attention and feedforward layers drive reasoning through opposing dynamics. These findings establish that the cost of thought is determined not by task difficulty but by manifold geometry - offering a blueprint for inference acceleration where topology permits.

The Geometry of Thought: How Scale Restructures Reasoning In Large Language Models

TL;DR

This paper shows that scaling large language models reshapes reasoning through domain-specific geometric reorganizations rather than a uniform performance boost. By analyzing over 25,000 chain-of-thought trajectories across Law, Science, Code, and Math and two model scales, it uncovers three phases—Crystallization in Law, Liquidity in Science/Math, and Lattice in Code—each with distinct effects on global dimension , intrinsic dimension , alignment, and clustering, plus a universal oscillatory coherence of about . It introduces Neural Reasoning Operators to map initial to terminal hidden states and demonstrates that favorable geometry predicts operator learnability, enabling amortized inference in crystalline domains. The findings imply that the cost of thought depends on manifold geometry and point toward geometry-aware deployment, compression, and safety considerations, with practical methods for operator-based inference and a reproducible measurement suite. Overall, the work provides a geometry-centric map of how scale restructures reasoning, offering a blueprint for domain-specific acceleration and more reliable deployment of high-stakes AI systems.

Abstract

Scale does not uniformly improve reasoning - it restructures it. Analyzing 25,000+ chain-of-thought trajectories across four domains (Law, Science, Code, Math) and two scales (8B, 70B parameters), we discover that neural scaling laws trigger domain-specific phase transitions rather than uniform capability gains. Legal reasoning undergoes Crystallization: 45% collapse in representational dimensionality (d95: 501 -> 274), 31% increase in trajectory alignment, and 10x manifold untangling. Scientific and mathematical reasoning remain Liquid - geometrically invariant despite 9x parameter increase. Code reasoning forms a discrete Lattice of strategic modes (silhouette: 0.13 -> 0.42). This geometry predicts learnability. We introduce Neural Reasoning Operators - learned mappings from initial to terminal hidden states. In crystalline legal reasoning, our operator achieves 63.6% accuracy on held-out tasks via probe decoding, predicting reasoning endpoints without traversing intermediate states. We further identify a universal oscillatory signature (coherence ~ -0.4) invariant across domains and scales, suggesting attention and feedforward layers drive reasoning through opposing dynamics. These findings establish that the cost of thought is determined not by task difficulty but by manifold geometry - offering a blueprint for inference acceleration where topology permits.
Paper Structure (62 sections, 11 equations, 10 figures, 1 table)

This paper contains 62 sections, 11 equations, 10 figures, 1 table.

Table of Contents

  1. Introduction
  2. The dynamical view: reasoning as trajectories
  3. A geometric measurement suite
  4. From geometry to operators: the amortization hypothesis
  5. Preview of findings: a phase diagram of thought
  6. Contributions
  7. Related Work
  8. Neural Scaling Laws
  9. Chain-of-Thought Reasoning
  10. Mechanistic Interpretability
  11. The Manifold Hypothesis
  12. Phase Transitions in Neural Networks
  13. Inference Acceleration
  14. Operator Learning
  15. Kolmogorov-Arnold Networks and Spectral Operator Architectures
  16. ...and 47 more sections

Figures (10)

  • Figure 1: The Topological Phase Diagram of Thought. Scatter plot of Alignment vs. Silhouette Score across $N > 25,000$ reasoning trajectories. Arrows indicate the 8B$\to$70B transition for each domain. Law (red) undergoes a phase transition from the Liquid region to the Crystalline region. Science (blue) and Math (orange) remain in the Liquid/Aligned region. Code (green) occupies a distinct Lattice region characterized by high clustering.
  • Figure 2: Scaling Laws Are Domain-Dependent. (A) Global dimensionality ($d_{95}$) across four domains at 8B and 70B scale. Law exhibits a 45% collapse from 501 to 274 dimensions; Math and Science remain invariant. (B) Percentage change in global dimension, with $\pm 5\%$ invariance zone. Law is a statistical outlier; Science and Math fall within the invariance zone.
  • Figure 3: The Crystallization Event vs. Scale Invariance. Slope chart comparing global dimension ($d_{95}$) between 8B and 70B models. Law (red) exhibits a steep $-45\%$ collapse; Science (blue) and Math (orange) remain flat, demonstrating scale invariance in exploratory domains.
  • Figure 4: Complete Geometric Metrics Matrix. Heatmap of six geometric properties across 4 domains $\times$ 2 scales. Darker green indicates higher structure. Note the dramatic increase in structure for Law 70B compared to Law 8B across alignment, silhouette, compactness, and G/L coherence metrics.
  • Figure 5: Manifold Modularity Across All Domains. Bar chart of Global/Local dimension ratio for all eight conditions. The green dashed line indicates $G/L = 1$ (uniform/flat manifold). Law exhibits $10\times$ untangling ($9.82\times \to 0.98\times$); Math remains highly modular ($9.82\times$) at both scales; Science shows partial convergence; Code maintains low G/L throughout.
  • ...and 5 more figures