Spectral Archaeology: The Causal Topology of Model Evolution

TL;DR

This work tackles the opacity of proprietary model training by introducing Spectral Archaeology, a training-free probe that converts layer-wise attention into token-graph spectra and computes $\lambda_2$, smoothness, and spectral entropy to reveal training-regime fingerprints. Through a Passive Voice Probe and a broad cross-model, cross-language analysis, it uncovers a curriculum-induced Passive-Triggered Connectivity Collapse (PTCC) confined largely to Layer 2–4 in English under code-to-chat transitions, with broader regulatory patterns across model families. The study identifies four recurrent processing strategies, demonstrates a mechanistic, localized cause in a sparse Layer-2 head patch, and shows partial recovery via targeted steering, distinguishing capacity restoration from semantic boosting. Beyond detecting brittleness, the work formalizes a forensic rule-based classifier and introduces a typological gearbox in Phi-4 that adapts topology to tokenization density, offering a practical, weight-only tool for auditing provenance, failure prediction, and training verification in opaque pipelines.

Abstract

Behavioral benchmarks tell us \textit{what} a model does, but not \textit{how}. We introduce a training-free mechanistic probe using attention-graph spectra. Treating each layer as a token graph, we compute algebraic connectivity ($λ_2$), smoothness, and spectral entropy. Across 12 models and 10 languages, these measures yield stable ``spectral fingerprints'' that expose discontinuities missed by standard evaluation. We report four results. (1) Models undergoing specific curriculum transitions (e.g., code-to-chat) show an English-only, syntax-triggered connectivity failure on non-canonical constructions, reaching $Δλ_2 \approx -0.76$. We term this scar \textit{Passive-Triggered Connectivity Collapse} (PTCC). Analysis of the Phi lineage reveals that PTCC appears and resolves across developmental stages, implicating brittle curriculum shifts rather than synthetic data per se. (2) PTCC reflects a specialization trade-off: strengthened formal routing at the expense of stylistic flexibility. (3) We identify four recurrent processing strategies; simple frozen-threshold rules enable perfect forensic identification across lineages. (4) Mechanistically, PTCC localizes to a sparse Layer 2 ``compensatory patch'' of heads that fails under syntactic stress; activation steering can partially restore connectivity, recovering $\approx 38\%$ of lost information flow. Finally, dominant topological regimes track tokenization density more than language identity, suggesting ``healthy'' geometry varies systematically across scripts. Overall, attention-graph spectra provide a practical tool for auditing and training-regime verification.

Figures (15)

• Figure 1: Spectral Archaeology of Model Evolution. We transform attention matrices into token graphs to detect the "Synthetic Scar" or PTCC, a catastrophic connectivity collapse ($\lambda_2 \downarrow$) triggered by syntactic stress. Our spectral fingerprints reveal latent discontinuities missed by behavioral benchmarks and enable forensic identification of model training regimes.
• Figure 2: Fiedler Value Differential. Layer-wise change in algebraic connectivity ($\Delta\lambda_2$) between passive and active constructions across languages; English exhibits the "synthetic scar".
• Figure 3: Spectral Mechanisms of Recovery: Connectivity vs Simplification. Metrics computed on an arbitrary natural sample ('The book was written by the man.'). The Active baseline (green) represents the ideal spectral profile. Structural steering (orange) restores Fiedler connectivity towards the Active profile. Chain-of-Thought (yellow) maximizes entropy but collapses structure, indicating an off-manifold processing pathway.
• Figure 4: Phi-1.5: Pre-Structural Fragmentation. Unlike Phi-3, English (green) does not collapse under passive voice because it never achieves high connectivity in the first place. Baseline English $\lambda_2 \approx 0.25$ is comparable to passive state, resulting in near-zero delta. Other languages show varied regimes, with some (Russian, Chinese) exhibiting high connectivity. This pattern suggests the model processes English through permanently fragmented pathways rather than specialized circuits that fracture under stress.
• Figure 5: Phi-2: Language-Conditioned Regime Separation. Phi-2 exhibits a striking bimodal distribution: some languages (Russian, Chinese, Japanese) achieve very high connectivity ($\lambda_2 > 0.85$), while English and French remain in a low-connectivity regime ($\lambda_2 \approx 0.20$). Passive voice produces minimal delta because the topology is already language-stratified rather than syntax-sensitive. This represents "structural apartheid" rather than syntactic vulnerability.