SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devices

TL;DR

SpecExec tackles the challenge of interactive LLM inference on consumer hardware by uniting RAM offloading with speculative execution. It deterministically builds a large draft-tree of likely continuations via a parallel SSSP (shortest-path) search and caches these continuations to verify with the target model in a single pass, preserving exact sampling distributions. Empirically, SpecExec delivers substantial speedups (up to 10–18x over sequential baselines) and enables 50B+ models to run on consumer GPUs with 4-bit quantization (4–6 tokens/s) or 16-bit weights (2–3 tokens/s). The approach demonstrates that careful drafting, caching, and verification can overcome bandwidth-bound limitations, making local interactive inference more practical on offloaded models. The work also provides guidance on draft-tree size, coverage, and hardware considerations for real-world deployment.

Abstract

As large language models gain widespread adoption, running them efficiently becomes crucial. Recent works on LLM inference use speculative decoding to achieve extreme speedups. However, most of these works implicitly design their algorithms for high-end datacenter hardware. In this work, we ask the opposite question: how fast can we run LLMs on consumer machines? Consumer GPUs can no longer fit the largest available models (50B+ parameters) and must offload them to RAM or SSD. When running with offloaded parameters, the inference engine can process batches of hundreds or thousands of tokens at the same time as just one token, making it a natural fit for speculative decoding. We propose SpecExec (Speculative Execution), a simple parallel decoding method that can generate up to 20 tokens per target model iteration for popular LLM families. It utilizes the high spikiness of the token probabilities distribution in modern LLMs and a high degree of alignment between model output probabilities. SpecExec takes the most probable tokens continuation from the draft model to build a "cache" tree for the target model, which then gets validated in a single pass. Using SpecExec, we demonstrate inference of 50B+ parameter LLMs on consumer GPUs with RAM offloading at 4-6 tokens per second with 4-bit quantization or 2-3 tokens per second with 16-bit weights.

Figures (11)

• Figure 1: Acceptance counts vs draft size (left), forward pass GPU time vs input size (right). Llama 2-7B draft model, offloaded Llama 2-70B target model, MTBench dataset, t=0.6 and top-p=0.9.
• Figure 2: Llama-2 70B Chat model cumulative probability of most likely tokens compared to the draft model choice (all Llama draft models are chat versions), OASST1 dataset.
• Figure 3: Generation rate depending on the draft budget size for Llama 2-7B Chat as the draft model and Llama 2-70B Chat as the target model, MTBench zheng2023judging dataset. Results are obtained with an A100 GPU.
• Figure 4: Generation rate depending on the draft budget size for Llama 2-7B Chat as the draft model and Llama 2-70B Chat as the target model, C4 dataset. Results are obtained with an A100 GPU.
• Figure 5: Acceptance counts (left) and generation speed (right) depending on the draft size. Llama 2-7B is used as the draft model, offloaded Llama 2-70B is the target model, MTBench dataset, t=0.6 and top-p=0.9. Results are obtained with an A100 GPU.