Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration

TL;DR

This work tackles inference latency in large language models by enhancing speculative decoding with a CTC-based draft model (CTC-drafter) that strengthens dependencies between draft tokens. By training a parallel Attention Draft Module with sequence-level CTC loss and applying a CTC Transform during verification, the method achieves higher draft acceptance rates and faster end-to-end inference across multiple base-model sizes and tasks. Empirical results on MT-bench and GSM8K show speedups typically above 2×, with strong transferability to LLaMA-2-Chat, indicating practical impact for real-time or multi-turn generation scenarios. The approach bridges draft-generation speed and quality, offering a general, transferable framework for context-aware drafting in speculative decoding.

Abstract

Inference acceleration of large language models (LLMs) has been put forward in many application scenarios and speculative decoding has shown its advantage in addressing inference acceleration. Speculative decoding usually introduces a draft model to assist the base LLM where the draft model produces drafts and the base LLM verifies the draft for acceptance or rejection. In this framework, the final inference speed is decided by the decoding speed of the draft model and the acceptance rate of the draft provided by the draft model. Currently the widely used draft models usually generate draft tokens for the next several positions in a non-autoregressive way without considering the correlations between draft tokens. Therefore, it has a high decoding speed but an unsatisfactory acceptance rate. In this paper, we focus on how to improve the performance of the draft model and aim to accelerate inference via a high acceptance rate. To this end, we propose a CTC-based draft model which strengthens the correlations between draft tokens during the draft phase, thereby generating higher-quality draft candidate sequences. Experiment results show that compared to strong baselines, the proposed method can achieve a higher acceptance rate and hence a faster inference speed.

Figures (4)

• Figure 1: Illustration of CTC-drafter model training and inference strategy.
• Figure 2: Average number of tokens accepted per decoding step in different question categories on MT-bench, with Vicuna-7B as base model. The performance on Vicuna-13B and Vicuna-33B is consistent with this result. The blue color represents CTC-drafter method, orange color represents Medusa method and green color represents baseline. All evaluation experiments are conducted on the same device.
• Figure 3: The percentage of time consumed for different processes based on CTC-drafter(left) and Medusa(right) speculation strategies. The "others" part mainly contains matrix operations involved in token tree verification.
• Figure 4: The bar charts of speedup ratio relative to vanilla method $\gamma$(top) and average number of tokens accepted per decoding step $\beta$(bottom) across different model types and sizes with CTC-drafter. The blue bar represents performance on MT-bench and the orange bar represents GSM8K. All evaluation experiments are conducted on the same devices.