Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding

TL;DR

The paper tackles the challenge of deploying large LLMs on memory-limited consumer GPUs by introducing Substitute Speculative Decoding (SubSpec), a lossless and training-free accelerator for parameter offloading. SubSpec builds a highly aligned, GPU-resident draft model using low-bit substitute weights for offloaded layers, coupled with shared GPU-resident layers and a shared KV-Cache, enabling deeper context-aware draft trees and high average acceptance lengths $\tau$. Theoretical analysis shows that in offloading scenarios maximizing $\tau$ yields the greatest speedups, which SubSpec achieves in practice by data-free quantization and careful draft-tree design, achieving up to $9.1$–$12.5\times$ speedups across MT-Bench, generation benchmarks, and reasoning tasks on GPUs with $8$–$24$ GB VRAM. This work demonstrates that large, high-quality LLMs can be effectively deployed locally on consumer hardware with substantial throughput gains, while outlining limitations (minimum memory, quantization trade-offs) and directions for extending the approach to other architectures and interconnect advances.

Abstract

The immense model sizes of large language models (LLMs) challenge deployment on memory-limited consumer GPUs. Although model compression and parameter offloading are common strategies to address memory limitations, compression can degrade quality, and offloading maintains quality but suffers from slow inference. Speculative decoding presents a promising avenue to accelerate parameter offloading, utilizing a fast draft model to propose multiple draft tokens, which are then verified by the target LLM in parallel with a single forward pass. This method reduces the time-consuming data transfers in forward passes that involve offloaded weight transfers. Existing methods often rely on pretrained weights of the same family, but require additional training to align with custom-trained models. Moreover, approaches that involve draft model training usually yield only modest speedups. This limitation arises from insufficient alignment with the target model, preventing higher token acceptance lengths. To address these challenges and achieve greater speedups, we propose SubSpec, a plug-and-play method to accelerate parameter offloading that is lossless and training-free. SubSpec constructs a highly aligned draft model by generating low-bit quantized substitute layers from offloaded target LLM portions. Additionally, our method shares the remaining GPU-resident layers and the KV-Cache, further reducing memory overhead and enhance alignment. SubSpec achieves a high average acceptance length, delivering 9.1x speedup for Qwen2.5 7B on MT-Bench (8GB VRAM limit) and an average of 12.5x speedup for Qwen2.5 32B on popular generation benchmarks (24GB VRAM limit).

Figures (6)

• Figure 1: Impact of draft model characteristics on speculative decoding performance, tested under the MT-Bench zheng2023judging benchmark. Left: Maximal speedup achieved by different draft models varies with draft depth in tree-based speculative decoding. Right: Average inference latency per token for Qwen2.5 7B (8GB GPU memory constraint) of different methods with optimal draft depths. SubSpec utilizes a higher draft model computation to minimize costly memory transfers of target model parameters.
• Figure 2: Comparative performance of Qwen2.5 7B using SD with different draft approaches across varying draft depths, tested under the MT-Bench benchmark.
• Figure 3: Left: Draft model architecture of SubSpec. SubSpec maintains additional low-bit substitute weights to keep the full draft model on the GPU. Right: Shared KV-Cache generation pipeline of SubSpec. The draft model reuses the KV-Cache of the target model to achieve better alignment and memory efficiency. This illustration serves as a simple sequential demonstration, while in practice, we maintain a flattened token tree for tree decoding.
• Figure 4: Demonstration of the false positive path issue (with k = 2 for simplification). Correct draft tokens may be dropped due to a lower cumulative probability score. The number in the parentheses in each token denotes the cumulative probability score.
• Figure 5: Execution timelines of the verification step for different decoding strategies