Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding

TL;DR

This work tackles the latency of large language model inference by enhancing speculative decoding with Hierarchy Drafting (HD). HD organizes multiple draft-token sources into a hierarchy of databases aligned with temporal locality and accesses them in order from the most local to broader sources, enabling lossless acceleration without parameter updates. Across Vicuna and Llama-2 models on Spec-Bench, HD delivers robust speedups across tasks and temperatures, outperforming single-source database methods and narrowing gaps to retraining-based approaches. The framework is plug-and-play, scalable with additional sources, and demonstrates practical potential for real-world, multi-model serving scenarios.

Abstract

Accelerating inference in Large Language Models (LLMs) is critical for real-time interactions, as they have been widely incorporated into real-world services. Speculative decoding, a fully algorithmic solution, has gained attention for improving inference speed by drafting and verifying tokens, thereby generating multiple tokens in a single forward pass. However, current drafting strategies usually require significant fine-tuning or have inconsistent performance across tasks. To address these challenges, we propose Hierarchy Drafting (HD), a novel lossless drafting approach that organizes various token sources into multiple databases in a hierarchical framework based on temporal locality. In the drafting step, HD sequentially accesses multiple databases to obtain draft tokens from the highest to the lowest locality, ensuring consistent acceleration across diverse tasks and minimizing drafting latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters demonstrate that HD outperforms existing database drafting methods, achieving robust inference speedups across model sizes, tasks, and temperatures.

Figures (10)

• Figure 1: Result of database drafting methods on Spec-Bench Spec_Survey with Vicuna-7B vicuna. The values in the plot denote the speedup against autoregressive decoding. (Left) QA and summarization task performance. (Right) Acceptance ratio and drafting latency.
• Figure 2: Overview of database drafting and our proposed method, Hierarchy Drafting (HD). A. Previous database drafting methods retrieve draft tokens from a single database constructed from a single source, leading to inconsistent acceleration gains across different scenarios. B. HD, however, leverages multiple databases encompassing diverse sources to improve token coverage, ensuring consistent performance. B-1. During the drafting process, databases are accessed sequentially from the smallest to the largest, based on the temporal locality of the token sequences. B-2. Multiple draft token sequences are verified in parallel, and the sequence with the highest number of accepted tokens is finally selected as the generated output.
• Figure 3: (Upper) 4-gram statistics for 100 generations of Llama-2-7b. The x-axis shows the order of 4-grams across 100 generations, with major ticks marking generation steps. The y-axis represents unique 4-gram indices. Red dots indicate 4-grams from previous processes, while blue dots represent those from the current process. (Lower) Frequency analysis of two 4-grams, represented by red and blue dots, respectively.
• Figure 4: (Left) Speedup comparison with non-database drafting methods with Vicuna-7B on Spec-Bench. (Right) Speedup comparison of database drafting methods across six tasks of Spec-Bench.
• Figure 5: (Left) Verify success and draft latency for the databases $\mathcal{D}_c$, $\mathcal{D}_m$, and $\mathcal{D}_s$ in HD. Verify success represents the proportion of accepted accesses relative to the total accesses. (Right) Verify success density plots for each database across six tasks in Spec-Bench. Both results are conducted on Spec-Bench by using Llama-2-7b.