Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree

TL;DR

Falcon tackles the latency-accuracy trade-off in LLM inference by introducing a semi-autoregressive speculative decoding framework that blends fast multi-token drafting with strong intra-block dependencies. It achieves this through Coupled Sequential Glancing Distillation (CSGD) and a custom-designed decoding tree, enabling multiple forward passes and parallel verification while preserving draft quality. Theoretical analysis links CSGD to improved token dependencies within a block, and extensive experiments show lossless speedups of $2.91\times$ to $3.51\times$ on Vicuna and LLaMA2-Chat with a compact two-layer-ish drafter, outperforming AR and SAR baselines such as Eagle, Medusa, Lookahead, SPS, and PLD. The approach offers practical impact for real-time LLM applications under resource constraints and lays groundwork for further improvements in SAR decoding via dynamic trees and stronger token interdependencies.

Abstract

Striking an optimal balance between minimal drafting latency and high speculation accuracy to enhance the inference speed of Large Language Models remains a significant challenge in speculative decoding. In this paper, we introduce Falcon, an innovative semi-autoregressive speculative decoding framework fashioned to augment both the drafter's parallelism and output quality. Falcon incorporates the Coupled Sequential Glancing Distillation technique, which fortifies inter-token dependencies within the same block, leading to increased speculation accuracy. We offer a comprehensive theoretical analysis to illuminate the underlying mechanisms. Additionally, we introduce a Custom-Designed Decoding Tree, which permits the drafter to generate multiple tokens in a single forward pass and accommodates multiple forward passes as needed, thereby boosting the number of drafted tokens and significantly improving the overall acceptance rate. Comprehensive evaluations on benchmark datasets such as MT-Bench, HumanEval, and GSM8K demonstrate Falcon's superior acceleration capabilities. The framework achieves a lossless speedup ratio ranging from 2.91x to 3.51x when tested on the Vicuna and LLaMA2-Chat model series. These results outstrip existing speculative decoding methods for LLMs, including Eagle, Medusa, Lookahead, SPS, and PLD, while maintaining a compact drafter architecture equivalent to merely two Transformer layers.

Figures (6)

• Figure 1: Framework of Falcon. It illustrates the computational process and displays the corresponding generation results of each forward pass for enhanced SAR drafting.
• Figure 2: The training procedure of CSGD. $\hat{Y}$ is the initial predicted feature representation sequence of the draft model, $Y^{t}$ is the ground-truth feature calculated by LLMs, $t_i$ is the original token, $t_i^t$ is the target token generated by LLMs, $h_i$ is the original feature sequence, and $h_i^t$ is the target feature generated by LLMs.
• Figure 3: SAR decoding tree attention illustrated. This visualization demonstrates that SAR tree attention is utilized to process multiple candidates in parallel, and k is set to 2.
• Figure 4: Speedup ratio of Vicuna and LLaMA2-Chat on MT-bench for greedy (temperature=0).
• Figure 5: The tree structure used for Vicuna series.