Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Shenggui Li, Chao Wang, Yikai Zhu, Yubo Wang, Fan Yin, Shuai Shi, Yefei Chen, Xiaomin Dong, Qiaoling Chen, Jin Pan, Ji Li, Laixin Xie, Yineng Zhang, Lei Yu, Yonggang Wen, Ivor Tsang, Tianwei Zhang

Abstract

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.

SpecForge: A Flexible and Efficient Open-Source Training Framework for Speculative Decoding

Abstract

Large language models incur high inference latency due to sequential autoregressive decoding. Speculative decoding alleviates this bottleneck by using a lightweight draft model to propose multiple tokens for batched verification. However, its adoption has been limited by the lack of high-quality draft models and scalable training infrastructure. We introduce SpecForge, an open-source, production-oriented framework for training speculative decoding models with full support for EAGLE-3. SpecForge incorporates target-draft decoupling, hybrid parallelism, optimized training kernels, and integration with production-grade inference engines, enabling up to 9.9x faster EAGLE-3 training for Qwen3-235B-A22B. In addition, we release SpecBundle, a suite of production-grade EAGLE-3 draft models trained with SpecForge for mainstream open-source LLMs. Through a systematic study of speculative decoding training recipes, SpecBundle addresses the scarcity of high-quality drafts in the community, and our draft models achieve up to 4.48x end-to-end inference speedup on SGLang, establishing SpecForge as a practical foundation for real-world speculative decoding deployment.
Paper Structure (25 sections, 6 equations, 6 figures, 4 tables, 3 algorithms)

This paper contains 25 sections, 6 equations, 6 figures, 4 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Speculative Decoding
  4. Theoretical Speedup
  5. Architecture Paradigm
  6. Training-Time Test
  7. Challenges
  8. $\mathtt{SpecForge}$
  9. Target-Draft Decoupling
  10. Hybrid Parallelism
  11. Computation Optimization
  12. Sparse Tree Attention
  13. Memory-Efficient Gradient Computation
  14. Evaluation of $\mathtt{SpecForge}$
  15. Experimental Setup
  16. ...and 10 more sections

Figures (6)

  • Figure 1: EAGLE3 attention mask used in Training-Time Testing.
  • Figure 2: Architecture comparisons
  • Figure 3: Training time with different execution backends
  • Figure 4: Comparison of execution time and memory usage between naive EAGLE3 attention and our optimized kernel.
  • Figure 5: Inference performance of Llama3.1-8B with EAGLE3 trained on datasets with and without regenerating the responses. The experiment was conducted on 1 H200 GPU with batch size 8.
  • ...and 1 more figures