Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds

TL;DR

The work analyzes how deep Transformer models exhibit emergent multi-step reasoning by treating the hidden-state trajectory as a flow on a latent manifold and identifying a phase-transition-like shift in spectral structure. By integrating a Renormalization Group perspective with random-matrix baselines and information-geometric diagnostics, the authors demonstrate that, beyond a critical depth γ_c ≈ 0.42, covariance spectra develop spikes, effective dimensionality collapses, and new, object-like Transient Class Objects (TCOs) form, stabilizing reasoning across steps. They formalize forward passes as coarse-graining maps, prove sufficient conditions for spectral collapse via transverse contraction and mixture-model structure, and validate predictions with multiple open-weight model families using activation covariances and LOP. The findings provide a principled link between logical separability, spectral decay, and low-rank latent structure, with implications for interpreting and guiding emergent reasoning in large language models. This framework highlights how depth acts as an intrinsic cooling mechanism that concentrates latent trajectories into reusable basins, informing both theoretical understanding and practical prompts for improved reasoning in AI systems.

Abstract

We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens. Treating the hidden-state trajectory as a flow on an implicit Riemannian manifold, we analyze the layerwise covariance spectrum of activations, where $C^{(\ell)}=\mathbb{E}[h^{(\ell)}h^{(\ell)\top}]$, and track deviations from a random-matrix bulk. Across model scales (1.5B--30B), we observe a sharp reduction in effective dimensionality consistent with a phase transition: an order parameter based on sparsity/localization, $Ω(h)=1-\|h\|_1/(\sqrt{d}\|h\|_2)$, exhibits a discontinuity near a critical normalized depth $γ_c\approx 0.42$ in sufficiently large models. We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics. The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space, which we call Transient Class Objects (TCOs). We provide theoretical conditions connecting logical separability to spectral decay and validate the predicted signatures with layerwise probes on multiple open-weight model families.

Figures (3)

• Figure 1: Microscopic density of Object Integrity $\Omega$ vs. Depth. A distinct high-integrity mode ($\Omega > 0.8$) emerges in reasoning-capable models (MiroThinker-30B, Fimbulvetr-11B), characterizing the onset of the "solid" phase. This bimodal separation contrasts with the unimodal, purely "liquid" dynamics observed in smaller baselines.
• Figure 2: Spectral anomaly via eigenvalues/SVD of $\widehat{C}^{(l)}$. Post-critical layers show tail suppression and/or spike separation, consistent with contraction and/or mixture-induced low-rank structure.
• Figure 3: Order parameter $m(\gamma)$ across depth. The steepest slope (or susceptibility peak) defines an operational $\widehat{\gamma}_c$.

Theorems & Definitions (19)

• Proposition 2.1: Basic bounds
• Proposition 2.2: Sharp bounds and extremizers
• proof
• Remark 1
• Definition 3.1: Hyperbolic embedding hypothesis (diagnostic)
• Proposition 4.1: Variational characterization of attention via free energy
• proof
• Definition 5.1: Marchenko--Pastur distribution
• Remark 2
• Theorem 6.1: Transverse contraction implies decay of transverse covariance