ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces

TL;DR

ThinkRouter tackles efficient reasoning by dynamically routing thinking between latent representations and discrete tokens based on inference-time confidence, specifically the maximum next-token probability $p_t^{\max}$ and a routing threshold $\tau$. It identifies that latent reasoning can be both efficient and fragile due to noise from aggregating low-confidence alternatives, and that routing to discrete space can curb this noise when confidence is low. Across STEM and coding benchmarks with diverse model scales, ThinkRouter yields substantial Pass@1 gains (up to $19.70$ points on average) while shortening generation length and calibrating errors from explicit CoT and latent reasoning. The method is training-free and inference-time, offering a practical, generalizable approach to robust, efficient reasoning in large language models, with deep analysis of confidence dynamics and termination behavior that explains the improvements.

Abstract

Recent work explores latent reasoning to improve reasoning efficiency by replacing explicit reasoning trajectories with continuous representations in a latent space, yet its effectiveness varies across settings. Analysis of model confidence dynamics under latent reasoning reveals that thinking trajectories ending in incorrect answers contain fewer low-confidence steps than those ending in correct answers. Meanwhile, we suggest that soft embeddings aggregated by multiple low-confidence thinking alternatives may introduce and propagate noise, leading to high confidence in unreliable reasoning trajectories. Motivated by these observations, ThinkRouter, an inference-time confidence-aware routing mechanism is proposed to avoid high confidence and noise for efficient reasoning. ThinkRouter routes thinking to the discrete token space when model confidence is low, and to the latent space otherwise. Extensive experiments on STEM reasoning and coding benchmarks across diverse large reasoning models demonstrate that ThinkRouter outperforms explicit CoT, random routing, and latent reasoning baselines in terms of accuracy, achieving an average improvement of 19.70 points in Pass@1, while reducing generation length by up to 15.55%. Further comprehensive analysis reveals that ThinkRouter can calibrate errors arising from explicit CoT and latent reasoning, and accelerates end-of-thinking token generation by globally lowering model confidence.

Figures (18)

• Figure 1: Ratio of low-confidence time steps ($p_t^{\max}<\tau$) within reasoning trajectories under Soft Thinking (latent-only reasoning) on GPQA Diamond. The incorrect predictions are associated with fewer low-confidence thinking time steps than the correct predictions.
• Figure 2: Overview of ThinkRouter. During thinking, when the maximum next-token probability $p_t^{\max}$ is lower than the routing threshold $\tau$, ThinkRouter routes thinking to the discrete token space; otherwise, ThinkRouter routes thinking to the latent space by calculating a probability-weighted soft embedding.
• Figure 3: Error calibration of ThinkRouter.
• Figure 4: Low-confidence time step ratio (%) across generation steps on Qwen3-8B with GPQA Diamond.
• Figure 5: $p_t^{\max}$ of last 10 time steps before the end-of-thinking token for Qwen3-8B.