Geometry of Reason: Spectral Signatures of Valid Mathematical Reasoning

TL;DR

This work introduces a training-free framework that detects valid mathematical reasoning in LLMs by analyzing attention as dynamic graphs through spectral diagnostics. Four metrics—Fiedler value, High-Frequency Energy Ratio, graph signal smoothness, and spectral entropy—consistently separate valid from invalid proofs across seven models from four families, achieving up to $d = 3.30$ and high calibration-based accuracy while requiring no training data. The method reveals that spectral signatures reflect logical coherence (Platonic validity) rather than compiler acceptance, and omits dependence on authorship style when properly controlled. An architectural dependency is shown: Sliding Window Attention shifts the discriminative signal toward late-layer Smoothness in Mistral, underscoring the need to consider attention topology in interpretation. These findings offer a principled, geometry-based tool for reasoning verification with immediate relevance to hallucination detection, AI safety, and proof-assistant workflows.

Abstract

We present a training-free method for detecting valid mathematical reasoning in large language models through spectral analysis of attention patterns. By treating attention matrices as adjacency matrices of dynamic graphs over tokens, we extract four interpretable spectral diagnostics, the Fiedler value (algebraic connectivity), high-frequency energy ratio (HFER), graph signal smoothness, and spectral entropy, that exhibit statistically significant differences between valid and invalid mathematical proofs. Experiments across seven transformer models from four independent architectural families (Meta Llama, Alibaba Qwen, Microsoft Phi, and Mistral AI) demonstrate that this spectral signature produces effect sizes up to Cohen's $d = 3.30$ ($p < 10^{-116}$), enabling 85.0--95.6\% classification accuracy under rigorous evaluation, with calibrated thresholds reaching 93--95\% on the full dataset. The method requires no training data, fine-tuning, or learned classifiers: a single threshold on a spectral metric suffices for high accuracy. Through systematic label correction, we discover that the spectral method detects logical coherence rather than compiler acceptance, identifying mathematically valid proofs that formal verifiers reject due to technical failures. We further identify an architectural dependency: Mistral-7B's Sliding Window Attention shifts the discriminative signal from HFER to late-layer Smoothness ($d = 2.09$, $p_{\text{MW}} = 1.16 \times 10^{-48}$), revealing that attention mechanism design affects which spectral features capture reasoning validity. These findings establish spectral graph analysis as a principled framework for reasoning verification with immediate applications to hallucination detection and AI safety monitoring.

Figures (19)

• Figure 1: Overview of the spectral analysis pipeline. Attention matrices from each transformer layer define dynamic token graphs; hidden states provide signals on these graphs. Spectral diagnostics capture properties of graph-signal interaction predictive of reasoning validity.
• Figure 2: The "Shape of Truth" (Llama-3.1-8B). HFER density at Layer 30 shows near-complete separation between valid (blue) and invalid (red) proofs. Effect size $d = 3.00$, $p_{\text{MW}} = 9.40 \times 10^{-64}$.
• Figure 3: Spectral Evolution of Reasoning. Llama-3.1-8B exhibits a phase transition in High-Frequency Energy Ratio (HFER). In early layers (0–10), valid and invalid proofs are indistinguishable. As reasoning deepens (Layers 15+), valid proofs (blue) maintain spectral smoothness, while hallucinations (red) disintegrate into high-frequency noise.
• Figure 4: Causal Link to Induction Mechanisms (Llama-3.1-8B). We ablated the top-$k$ induction heads identified via in-context copying scores. Top Left: The Fiedler Value (graph connectivity) significantly increases (worsens) in the critical pre-computation layers (Layers 4--10) as heads are deactivated, confirming that spectral smoothness is physically maintained by active induction circuits. Layer 12 Crossover: All metrics synchronize extrema at Layer 12 (Global Minima for Fiedler/Entropy, Global Maximum for HFER), marking the functional "decision boundary" where unstructured context crystallizes into logical token selection. The sharp HFER peak for the valid baseline (Blue) indicates a precise, low-entropy attention spike compared to the diffuse attention of ablated models.
• Figure 5: Architectural Determinism of Validity. Comparing Llama-3.1-8B (Global Attention) and Mistral-7B (Sliding Window Attention). Llama separates proofs via frequency (HFER), while Mistral separates them via local connectivity (Smoothness). This confirms that the spectral signature depends on the specific topology of the attention mechanism.

Theorems & Definitions (16)

• Definition 1: Dirichlet Energy
• Definition 2: High-Frequency Energy Ratio
• Definition 3: Spectral Entropy
• Definition 4: Fiedler Value
• Definition 5: Smoothness
• Example 6: Platonic vs. Compiler Validity
• Definition 7: Combinatorial Laplacian
• Definition 8: Normalized Laplacians
• Proposition 9: Laplacian Properties
• proof