The Geometric Reasoner: Manifold-Informed Latent Foresight Search for Long-Context Reasoning

TL;DR

The Geometric Reasoner (TGR) tackles the challenge of robust long-context reasoning under strict memory limits by offering a training-free inference-time framework. It performs manifold-informed latent foresight search in chunked generations, using unit-sphere state anchors, tangent-space candidate exploration, and soft geometric penalties (foresight value, bumpiness, and uniformity) to select anchors that steer subsequent chunks while keeping memory usage linear in chunk length. Empirically, TGR-Latent improves AUC on math and code benchmarks (e.g., up to 13-point AUC gains on Qwen3-8B) with modest overhead (about 1.1–1.3x tokens) compared to strong baselines, and ablations show the critical role of look-ahead value and diversity regularization in achieving robust coverage. The work demonstrates that training-time costs can be traded for inference-time geometric guidance, enabling scalable long-horizon reasoning without retuning model weights, with implications for more reliable programming assistants and math problem solvers under fixed budgets.

Abstract

Scaling test-time compute enhances long chain-of-thought (CoT) reasoning, yet existing approaches face a fundamental trade-off between computational cost and coverage quality: either incurring high training expense or yielding redundant trajectories. We introduce The Geometric Reasoner (TGR), a training-free framework that performs manifold-informed latent foresight search under strict memory bounds. At each chunk boundary, TGR scores candidate latent anchors via a lightweight look-ahead estimate combined with soft geometric regularizers that encourage smooth trajectories and diverse exploration. Chunk-wise KV cache resets keep memory linear in chunk length. On challenging math and code benchmarks, TGR improves robust trajectory coverage, measured by the area under the Pass@$k$ curve (AUC), by up to 13 points on Qwen3-8B, with negligible overhead of about 1.1--1.3 times.

Figures (5)

• Figure 1: TGR-Latent consistently outperforms baselines on Qwen3-8B. Manifold-informed latent foresight search steering converts modest inference-time compute into robust coverage without weight updates.
• Figure 2: Overview of reasoning frameworks. Unlike (a) test-time sampling, exploring trajectories without explicit structure, or (b) reinforcement learning, which internalizes preferences through costly training, (c) TGR introduces a training-free inference-time search over the latent manifold. It selects optimal chunk-level anchors via a soft geometric score combining foresight, bumpiness, and uniformity, then injects them into generation with KV cache resets to maintain linear memory cost.
• Figure 3: TGR dominates the inference efficiency frontier.Left: Pass@$k$ curves on MATH500 reveal that TGR-Latent sustains marginal gains beyond $k=32$ where baselines plateau. Middle & Right: On the cost--robustness plane, TGR-Latent occupies the upper-left corner, achieving the highest AUC at moderate token cost on both math and code benchmarks.
• Figure 4: Left: Latent-space mode diversity. RL-tuned baselines collapse into a unimodal cone, while TGR preserves a well-dispersed distribution, capturing a fuller range of valid reasoning paths. Right: Hyperparameter robustness. AUC increases with rollout depth $s$ and beam width $K$, but with diminishing returns.
• Figure 5: Training stage modulates inference-time controllability. TGR-Latent yields substantially larger gains on the SFT model (top), while improvement narrows after RL optimization (bottom), suggesting that inference-time search benefits models whose trajectory distribution retains residual flexibility.