SpecRouter: Adaptive Routing for Multi-Level Speculative Decoding in Large Language Models

TL;DR

SpecRouter reframes LLM inference as adaptive multi-hop routing across a heterogeneous pool of draft and verifier models to reduce latency while preserving the target output quality. It introduces a central ChainRouter, a ModelChainScheduler for dynamic chain selection, multi-level collaborative verification, and a StateManager with a logical validity mask to maintain KV-cache consistency during asynchronous, batched speculation. The system jointly optimizes draft/verification chains using real-time performance profiling and predictive similarity metrics (based on $\text{DTV}$ between model outputs) to minimize $T_{\text{eff}}$, the effective latency per generated token. Experimental results on datasets such as GSM8K, HumanEval, MT-Bench, and MGSM demonstrate that SpecRouter can outperform static speculative decoding approaches, achieving higher speedups (e.g., $EAF$ improvements) while maintaining output identical to the target model under deterministic decoding.

Abstract

Large Language Models (LLMs) present a critical trade-off between inference quality and computational cost: larger models offer superior capabilities but incur significant latency, while smaller models are faster but less powerful. Existing serving strategies often employ fixed model scales or static two-stage speculative decoding, failing to dynamically adapt to the varying complexities of user requests or fluctuations in system performance. This paper introduces \systemname{}, a novel framework that reimagines LLM inference as an adaptive routing problem solved through multi-level speculative decoding. \systemname{} dynamically constructs and optimizes inference "paths" (chains of models) based on real-time feedback, addressing the limitations of static approaches. Our contributions are threefold: (1) An \textbf{adaptive model chain scheduling} mechanism that leverages performance profiling (execution times) and predictive similarity metrics (derived from token distribution divergence) to continuously select the optimal sequence of draft and verifier models, minimizing predicted latency per generated token. (2) A \textbf{multi-level collaborative verification} framework where intermediate models within the selected chain can validate speculative tokens, reducing the verification burden on the final, most powerful target model. (3) A \textbf{synchronized state management} system providing efficient, consistent KV cache handling across heterogeneous models in the chain, including precise, low-overhead rollbacks tailored for asynchronous batch processing inherent in multi-level speculation. Preliminary experiments demonstrate the validity of our method.

Figures (3)

• Figure 1: Architecture of SpecRouter
• Figure 2: Dynamic model chain selection example. Based on profiled times ($T_i$) and acceptance probabilities ($\alpha_i$, derived from SimScore), the scheduler predicts the effective time per target token ($T_{\text{eff}}$) for candidate chains (using logic based on Eq. \ref{['eq:cascade_time_revised']}, conceptualized in Alg. \ref{['alg:chain_opt_simplified']}'s prediction function) and selects the chain with the minimum predicted time.
• Figure 3: Illustration of the logical cache_mask decoupling validity from physical KV cache storage. Invalid items (Mask=0) are ignored during attention, enabling correct handling of asynchronous progress after rollbacks without immediate data movement.