Student Guides Teacher: Weak-to-Strong Inference via Spectral Orthogonal Exploration

TL;DR

This paper identifies a geometric bottleneck in long-horizon reasoning by showing that hidden-state covariance collapses to a low-rank Bias Manifold, restricting access to high-value solutions in the Null Space. It proposes Spectral Orthogonal Exploration (SOE), a post-hoc, geometry-driven intervention that uses a weaker Student as an Orthogonal Probe to inject directions orthogonal to the Teacher's dominant subspace via Orthogonal Latent Stitching, thereby expanding the inference space. The method combines an Unbiased Covariance Estimator with Micro-SVD to estimate the bias manifold and selects probes by maximizing the orthogonal residual, achieving substantial gains in accuracy (average +$62.4\%$) and sampling efficiency (average +$113.7\%$) on challenging mathematical benchmarks. The results suggest that moving beyond probabilistic token sampling toward targeted geometric interventions can effectively overcome reasoning plateaus, offering a path toward more reliable advanced reasoning in LLMs, with acknowledged costs in computation and applicability to open-model settings.

Abstract

While Large Language Models (LLMs) demonstrate near-human capabilities, they often suffer from "Reasoning Collapse" in complex mathematical proving and long-horizon planning. Models tend to degenerate into low-rank Bias Manifold, where stochastic sampling merely produces lexical variations of erroneous logic rather than semantic exploration. This geometric collapse renders the model "blind" to high-value solutions that lie within its Null Space. To address this, we propose Spectral Orthogonal Exploration (SOE), a geometric framework operating on a counter-intuitive "Student Guides Teacher" paradigm. Specifically, we utilize a weak auxiliary agent not for imitation, but as an orthogonal probe. By explicitly navigating the Teacher's Null Space, SOE serves as a geometric bridge, effectively ejecting the model from local optima to explore diverse, high-value solution spaces. Experiments on mathematical benchmarks demonstrate that, relative to baseline methods, our approach improves average accuracy by 62.4% and increases average sampling efficiency by 113.7%, indicating a promising path toward overcoming performance plateaus in advanced reasoning tasks.

Figures (10)

• Figure 1: Geometric Interpretation of Reasoning Collapse. We characterize reasoning collapse as the transition of the state space from a high-dimensional Healthy Reasoning Manifold to a low-rank Bias Manifold. This confinement renders high-value solutions in the Null Space geometrically inaccessible, as the trajectory is trapped on a low-dimensional plane.
• Figure 1: Pass@16 accuracy on the Difficult Subset (problems where Teacher's greedy decoding failed). SOE demonstrates a consistent improvement over the Baseline, confirming that directed orthogonal exploration is superior to random perturbation. All the results are sampled under identical conditions, with same prompt, $temperature = 0.7$ and a maximum context (sampling) length capped at 8192 tokens.
• Figure 2: Mechanism of Spectral Orthogonal Exploration (SOE). To counteract space narrowing, we introduce an Orthogonal Probe as a geometric intervention. This force effectively disrupts the low-rank confinement and diversifies the reasoning trajectory, expanding the hyper-space to access high-quality solutions previously hidden in the teacher's null space.
• Figure 3: Universality of Reasoning Collapse across Datasets. The dynamics of Effective Rank during the reasoning process on AIME 24, AIME 25, Olympiad Bench and Omni Math. Across all benchmarks, we observe a consistent spectral degeneration pattern: as the token steps increase, the effective rank of the covariance matrix decays, indicating the model is collapsing into a low-dimensional bias manifold regardless of the problem difficulty.
• Figure 4: The SOE Framework

Theorems & Definitions (2)

• proof
• proof