Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning

TL;DR

STEER tackles the high inference cost of reasoning with large LLMs by introducing a domain-agnostic, external-model-free, stepwise routing framework that uses internal logit-based confidence from a smaller model calibrated by a two-component Gaussian Mixture Model to decide when to invoke a larger model. At each reasoning step, the framework computes a step-level confidence, estimates a posterior legitimacy for confidence via EM-fitted GMM components, and switches between models using a threshold. The approach yields strong cost-accuracy trade-offs across mathematical reasoning, multi-hop QA, and planning benchmarks, reducing FLOPs by up to 48% while maintaining or improving accuracy (e.g., +20% on AIME) and showing robustness across backbone models. It outperforms external-router baselines and LLM-Judge prompting, and its lightweight confidence routing enables scalable deployment without extra data collection or model training. The results suggest model-internal confidence is a robust, domain-agnostic signal for efficient inference in diverse reasoning tasks.

Abstract

Recent advances in Large Language Models (LLMs) - particularly model scaling and test-time techniques - have greatly enhanced the reasoning capabilities of language models at the expense of higher inference costs. To lower inference costs, prior works train router models or deferral mechanisms that allocate easy queries to a small, efficient model, while forwarding harder queries to larger, more expensive models. However, these trained router models often lack robustness under domain shifts and require expensive data synthesis techniques such as Monte Carlo rollouts to obtain sufficient ground-truth routing labels for training. In this work, we propose Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning (STEER), a domain-agnostic framework that performs fine-grained, step-level routing between smaller and larger LLMs without utilizing external models. STEER leverages confidence scores from the smaller model's logits prior to generating a reasoning step, so that the large model is invoked only when necessary. Extensive evaluations using different LLMs on a diverse set of challenging benchmarks across multiple domains such as Mathematical Reasoning, Multi-Hop QA, and Planning tasks indicate that STEER achieves competitive or enhanced accuracy while reducing inference costs (up to +20% accuracy with 48% less FLOPs compared to solely using the larger model on AIME), outperforming baselines that rely on trained external modules. Our results establish model-internal confidence as a robust, domain-agnostic signal for model routing, offering a scalable pathway for efficient LLM deployment.

Figures (5)

• Figure 1: STEER uses model confidence to route between the smaller LLM and the larger LLM in a step-wise fashion to achieve a good balance between performance and inference costs. More specifically, the larger LLM is invoked for the generation of the next step only if the smaller model is unconfident in generating the next step.
• Figure 2: Distributions of the confidence scores on correct and wrong reasoning trajectories for MATH500 on Gemma-3-Instruct 4B. The means of correct and wrong cases are shown with respective colors.
• Figure 3: MATH500 accuracy for percentile routing with other measures of token confidence. QwenMath denotes Qwen2.5-Math-Instruct. Gemma denotes Gemma3-Instruct.
• Figure 4: Overall latency induced by different methods, measured on the MATH500 benchmark.
• Figure 5: Ratio of steps using large models, by relative step position and difficulty in OmniMath using Gemma3-Instruct and Qwen2.5-Math-Instruct models. Relative step 0.1 denotes the first step, and 1.0 denotes the final step. Difficulty of 1.0 denotes the easiest cases, and 10.0 the hardest.