Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts

Wenhao Zeng, Xuteng Zhang, Yuling Shi, Chao Hu, Yuting Chen, Beijun Shen, Xiaodong Gu

TL;DR

GlimpRouter tackles the latency of large reasoning models by introducing a training-free, step-wise collaboration framework that uses the initial token's entropy to decide whether a reasoning step should be generated by a lightweight model or escalated to a large model. The core idea, inspired by the Aha Moment, is that the initial token's uncertainty effectively signals step difficulty, enabling a cost-efficient Probe-then-Dispatch routing mechanism. Empirical results on benchmarks like AIME25 show that GlimpRouter achieves a substantial accuracy boost (≈10.7%) with notable latency reductions (≈25.9%) relative to a standalone large model, and it remains orthogonal to token-level speculative decoding for compounded speedups. The approach establishes a simple, robust Pareto frontier for efficient reasoning, with practical implications for deploying LRMs in latency-constrained settings.

Abstract

Large Reasoning Models (LRMs) achieve remarkable performance by explicitly generating multi-step chains of thought, but this capability incurs substantial inference latency and computational cost. Collaborative inference offers a promising solution by selectively allocating work between lightweight and large models, yet a fundamental challenge remains: determining when a reasoning step requires the capacity of a large model or the efficiency of a small model. Existing routing strategies either rely on local token probabilities or post-hoc verification, introducing significant inference overhead. In this work, we propose a novel perspective on step-wise collaboration: the difficulty of a reasoning step can be inferred from its very first token. Inspired by the "Aha Moment" phenomenon in LRMs, we show that the entropy of the initial token serves as a strong predictor of step difficulty. Building on this insight, we introduce GlimpRouter, a training-free step-wise collaboration framework. GlimpRouter employs a lightweight model to generate only the first token of each reasoning step and routes the step to a larger model only when the initial token entropy exceeds a threshold. Experiments on multiple benchmarks demonstrate that our approach significantly reduces inference latency while preserving accuracy. For instance, GlimpRouter attains a substantial 10.7% improvement in accuracy while reducing inference latency by 25.9% compared to a standalone large model on AIME25. These results suggest a simple yet effective mechanism for reasoning: allocating computation based on a glimpse of thought rather than full-step evaluation.

GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughts

TL;DR

GlimpRouter tackles the latency of large reasoning models by introducing a training-free, step-wise collaboration framework that uses the initial token's entropy to decide whether a reasoning step should be generated by a lightweight model or escalated to a large model. The core idea, inspired by the Aha Moment, is that the initial token's uncertainty effectively signals step difficulty, enabling a cost-efficient Probe-then-Dispatch routing mechanism. Empirical results on benchmarks like AIME25 show that GlimpRouter achieves a substantial accuracy boost (≈10.7%) with notable latency reductions (≈25.9%) relative to a standalone large model, and it remains orthogonal to token-level speculative decoding for compounded speedups. The approach establishes a simple, robust Pareto frontier for efficient reasoning, with practical implications for deploying LRMs in latency-constrained settings.

Abstract

Large Reasoning Models (LRMs) achieve remarkable performance by explicitly generating multi-step chains of thought, but this capability incurs substantial inference latency and computational cost. Collaborative inference offers a promising solution by selectively allocating work between lightweight and large models, yet a fundamental challenge remains: determining when a reasoning step requires the capacity of a large model or the efficiency of a small model. Existing routing strategies either rely on local token probabilities or post-hoc verification, introducing significant inference overhead. In this work, we propose a novel perspective on step-wise collaboration: the difficulty of a reasoning step can be inferred from its very first token. Inspired by the "Aha Moment" phenomenon in LRMs, we show that the entropy of the initial token serves as a strong predictor of step difficulty. Building on this insight, we introduce GlimpRouter, a training-free step-wise collaboration framework. GlimpRouter employs a lightweight model to generate only the first token of each reasoning step and routes the step to a larger model only when the initial token entropy exceeds a threshold. Experiments on multiple benchmarks demonstrate that our approach significantly reduces inference latency while preserving accuracy. For instance, GlimpRouter attains a substantial 10.7% improvement in accuracy while reducing inference latency by 25.9% compared to a standalone large model on AIME25. These results suggest a simple yet effective mechanism for reasoning: allocating computation based on a glimpse of thought rather than full-step evaluation.
Paper Structure (40 sections, 3 equations, 4 figures, 7 tables, 1 algorithm)

This paper contains 40 sections, 3 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. A Preliminary Study
  3. Study Design
  4. Step Entropy ($\mathbf{H}_\text{step}$).
  5. Initial Token Entropy ($\mathbf{H}_\text{init}$).
  6. Step-wise Perplexity ($\mathbf{PPL}_\text{step}$).
  7. LLM-as-a-Judge.
  8. Results and Analysis
  9. Information Distribution within Reasoning Steps.
  10. Correlations to Model Routing.
  11. Methodology
  12. Problem Formulation
  13. Overview
  14. Glimpse: Initial Token Probing
  15. Dynamic Model Routing
  16. ...and 25 more sections

Figures (4)

  • Figure 1: Comparison of the distributions of different uncertainty metrics. Unlike other metrics, which exhibit a generic unimodal distribution, the entropy of initial thought tokens ($\mathbf{H}_\text{init}$) displays a distinct bimodal and heavy-tailed distribution. This indicates that $\mathbf{H}_\text{init}$ serves as a discriminative signal, effectively capturing the "Aha Moments" that distinguish complex reasoning steps from routine derivations.
  • Figure 2: Alignment between the outputs generated by small and large models under various $\mathbf{H}_\text{init}$ intervals.
  • Figure 3: The overall framework of GlimpRouter. The collaborative inference pipeline operates in a step-wise manner. At the onset of each reasoning step, the SLM first generates a "glimpse" (the initial token $T_k$). GlimpRouter then computes the entropy $\mathbf{H}_\text{init}$ of this token to gauge step difficulty. Based on the threshold, the system dynamically routes the generation: (1) Delegate (Low $\mathbf{H}_\text{init}$): Routine steps are fully generated by the efficient SLM. (2) Intervene (High $\mathbf{H}_\text{init}$): Complex steps signaling cognitive pivots are routed to the capable LLM for robust reasoning. Finally, the LLM generates the conclusive answer based on the aggregated collaborative chain.
  • Figure 4: Sensitivity analysis of collaboration threshold on multiple benchmarks. GlimpRouter (red circles) establishes a superior Pareto frontier compared to the SpecReason (blue triangles). Dashed lines represent the performance of standalone models.