Dynamic Mix Precision Routing for Efficient Multi-step LLM Interaction

TL;DR

This work tackles the high compute cost of long-horizon LLM agentics by introducing a step-level dynamic mix-precision router that selects between high-precision and quantized models at each decision step. It pairs KL-divergence–based supervised supervision (KL-ST) with Group-Relative Policy Optimization (GRPO) to balance task success and inference cost, enabling efficient use of full-precision reasoning only at precision-critical steps. The approach is validated on ALFWorld across multiple model families and quantization levels, demonstrating superior accuracy-cost trade-offs and high gain per high-precision call (GHC) compared to baselines. By showing that a lightweight two-layer Transformer router suffices to guide precision decisions, the method offers practical, scalable improvements for deploying long-horizon LLM agents in real-world settings.

Abstract

Large language models (LLM) achieve strong performance in long-horizon decision-making tasks through multi-step interaction and reasoning at test time. While practitioners commonly believe a higher task success rate necessitates the use of a larger and stronger LLM model, multi-step interaction with a large LLM incurs prohibitive inference cost. To address this problem, we explore the use of low-precision quantized LLM in the long-horizon decision-making process. Based on the observation of diverse sensitivities among interaction steps, we propose a dynamic mix-precision routing framework that adaptively selects between high-precision and low-precision LLMs at each decision step. The router is trained via a two-stage pipeline, consisting of KL-divergence-based supervised learning that identifies precision-sensitive steps, followed by Group-Relative Policy Optimization (GRPO) to further improve task success rates. Experiments on ALFWorld demonstrate that our approach achieves a great improvement on accuracy-cost trade-off over single-precision baselines and heuristic routing methods.

Figures (8)

• Figure 1: Reasoning Case Study. Compared to the high precision model, the low precision model failed at certain critical steps. Our router successfully judges the difficulty of each step and provide an efficient combination of model executions that succeeds the task with a fraction of high precision model usage.
• Figure 2: Overview of our dynamic mix-precision routing framework. (a) Observation: From the failed reasoning trajectories of quantized LLMs, we observe a step-wise diversity in sensitivity against model quantization: The quantized model will likely to encounter some "critical steps" that is out of its capability, which leads to wrong reasoning outcome in the end. (b) Router training pipeline: We collect high-precision successful rollouts and derive binary supervision signals from KL divergence to train a lightweight two-layers Transformer router that predicts whether a step is precision-sensitive. The router is further refined using group-relative policy optimization (GRPO) with a cost-aware reward to balance task success and inference efficiency. (c) At inference time, the router dynamically selects between a high-precision LLM and a quantized LLM at each decision step.
• Figure 3: Step-level routing architecture with masked position-aware representations. The task description and each trajectory step (action–observation pair) are encoded by a frozen text encoder, producing a sequence of step embeddings. Learnable position embeddings are added to inject step-order information, while a binary mask is applied to ignore invalid or truncated steps in variable-length trajectories. A transformer encoder processes the masked sequence, and the hidden state of the last valid step is used to predict.
• Figure 4: Distribution of step-wise KL divergence of 200 episodes between low-precision(3bit) and high-precision(bf16) policies of Qwen-8B on Alfworld. The distribution is highly skewed and bimodal-like: most decision steps exhibit near-zero divergence, while a small fraction forms a distant tail with substantially larger divergence values, with very few samples in between. In experiment, we choose 85% as the alpha of the model.
• Figure 5: Methodology ablation comparing KL-ST and KL-ST + GRPO routing on Qwen3-8B and Qwen3-4B; GRPO-only is evaluated on Qwen3-8B. Success rate is plotted against high-precision usage ratio. Solid curves show the baseline trade-off frontiers for Qwen3-8B (blue) and Qwen3-4B (red) with random routing, while markers denote different routing strategies. The GHC metric is visualized as the slope of the line connecting the low-precision model to each method.