Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Serving

TL;DR

Nightjar tackles the inefficiency of fixed speculative-length in speculative decoding when serving large language models under dynamic request rates. It introduces a learning-based adaptive speculative decoding algorithm based on a contextual bandit with per-batch state and a cost-aware objective expressed as $1/g_bar_B_gamma + I(gamma_{t-1}=0 and gamma_t>0) c_prefill/gamma_t$, coupled with a moving-average estimate of goodput $g_bar_B_gamma$. The approach uses a three-level hierarchy of blocks bins and rounds and employs an offline lookup table for the prefill cost, achieving up to 14.8% throughput gains and 20.2% latency reductions on real-world traces. These results demonstrate practical benefits for real-time LLM serving under dynamic workloads and show that adaptive speculative-length selection can outperform static and other dynamic baselines.

Abstract

Speculative decoding (SD) accelerates LLM inference by verifying draft tokens in parallel. However, this method presents a critical trade-off: it improves throughput in low-load, memory-bound systems but degrades performance in high-load, compute-bound environments due to verification overhead. Current SD implementations use a fixed speculative length, failing to adapt to dynamic request rates and creating a significant performance bottleneck in real-world serving scenarios. To overcome this, we propose Nightjar, a novel learning-based algorithm for adaptive speculative inference that adjusts to request load by dynamically selecting the optimal speculative length for different batch sizes and even disabling speculative decoding when it provides no benefit. Experiments show that Nightjar achieves up to 14.8% higher throughput and 20.2% lower latency compared to standard speculative decoding, demonstrating robust efficiency for real-time serving.

Figures (11)

• Figure 1: Schematic roofline illustration of autoregressive decoding (w/o SD) and speculative decoding (SD); positions are illustrative.
• Figure 2: Throughput under different request loads and rates. $\gamma$ denotes the speculative length (tokens generated by the draft model per step). Note: Configurations with $\gamma \geq 4$ are omitted at high QPS due to GPU out-of-memory (OOM) errors.
• Figure 3: An overview of our adaptive approach with multi-armed bandit (MAB) for dynamic request load.
• Figure 4: Illustration of the Nightjar algorithm's three-level hierarchy. Each batch size maintains its own Epoch. Each epoch contains Blocks ($j_B$) of exponentially growing duration, which are responsible for the long-term schedule. These blocks are further divided into fixed-size Bins ($b_B$), the basic units for controlling the exploration-exploitation trade-off. Each round ($\tau_B$) is a decoding step.
• Figure 5: Method Comparison at Low Request Rate.