Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing

Yushu Zhao, Yubin Qin, Yang Wang, Xiaolong Yang, Huiming Han, Shaojun Wei, Yang Hu, Shouyi Yin

TL;DR

SPEQ addresses the latency bottleneck in large language model decoding by combining a lossless speculative decoding pipeline with a novel bit-sharing BSFP quantization that derives a 4-bit draft from the full FP16 model. A reconfigurable PE array co-optimizes the forward passes for both the draft and verification steps, enabling high hardware utilization and zero KV-cache overhead. The approach achieves substantial speedups (average around 2×) over FP16 and lossy baselines like Olive and Tender across 15 LLMs and multiple tasks, with an average draft-accept rate near 0.98 and no training or extra memory requirements. This algorithm-hardware co-design offers a practical pathway to real-time LLM inference by delivering lossless acceleration without compromising accuracy or safety.

Abstract

Large language models achieve impressive performance across diverse tasks but exhibit high inference latency due to their large parameter sizes. While quantization reduces model size, it often leads to performance degradation compared to the full model. Speculative decoding remains lossless but typically incurs extra overheads. We propose SPEQ, an algorithm-hardware co-designed speculative decoding method that uses part of the full-model weight bits to form a quantized draft model, thereby eliminating additional training or storage overhead. A reconfigurable processing element array enables efficient execution of both the draft and verification passes. Experimental results across 15 LLMs and tasks demonstrate that SPEQ achieves speedups of 2.07x, 1.53x, and 1.45x compared over FP16, Olive, and Tender, respectively.

From Quarter to All: Accelerating Speculative LLM Decoding via Floating-Point Exponent Remapping and Parameter Sharing

TL;DR

SPEQ addresses the latency bottleneck in large language model decoding by combining a lossless speculative decoding pipeline with a novel bit-sharing BSFP quantization that derives a 4-bit draft from the full FP16 model. A reconfigurable PE array co-optimizes the forward passes for both the draft and verification steps, enabling high hardware utilization and zero KV-cache overhead. The approach achieves substantial speedups (average around 2×) over FP16 and lossy baselines like Olive and Tender across 15 LLMs and multiple tasks, with an average draft-accept rate near 0.98 and no training or extra memory requirements. This algorithm-hardware co-design offers a practical pathway to real-time LLM inference by delivering lossless acceleration without compromising accuracy or safety.

Abstract

Large language models achieve impressive performance across diverse tasks but exhibit high inference latency due to their large parameter sizes. While quantization reduces model size, it often leads to performance degradation compared to the full model. Speculative decoding remains lossless but typically incurs extra overheads. We propose SPEQ, an algorithm-hardware co-designed speculative decoding method that uses part of the full-model weight bits to form a quantized draft model, thereby eliminating additional training or storage overhead. A reconfigurable processing element array enables efficient execution of both the draft and verification passes. Experimental results across 15 LLMs and tasks demonstrate that SPEQ achieves speedups of 2.07x, 1.53x, and 1.45x compared over FP16, Olive, and Tender, respectively.
Paper Structure (19 sections, 4 equations, 9 figures, 4 tables, 1 algorithm)

This paper contains 19 sections, 4 equations, 9 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background
  3. LLM Quantization
  4. Speculative Decoding
  5. SPEQ Algorithm
  6. Bit-level Underutilization in LLM Weights
  7. Bit-sharing Quantization
  8. Optimization for Speculative Decoding
  9. SPEQ Hardware Accelerator
  10. Overview
  11. Decoder for BSFP
  12. Reconfigurable PE Array
  13. Evaluation
  14. Experiment Setup
  15. LLM Benchmarks
  16. ...and 4 more sections

Figures (9)

  • Figure 1: Illustration of the speculative decoding process: a draft model generates multiple draft tokens, and a target model verifies the draft tokens in parallel.
  • Figure 2: Overview of SPEQ, an algorithm-hardware co-design approach for accelerating LLM speculative decoding.
  • Figure 3: Illustration of remapping: the first row "Original" means the binary encoding and real value of FP16 exponent. "Naive" means using naive E3M0 quantization, the two neighbor values are rounded to the same quantized value. The bits in bold mean the binary representation of the quantized value. "Remap" means remapping the binary encoding to create unique representations for 9 and 11, while 0-3 and 4-7 are rounded to the same quantized value. The wasted bit is set to 1 if the bit representation of the remapped value is different from the original value.
  • Figure 4: Overview of SPEQ hardware accelerator. The reconfigurable PE is capable of processing both the quantized GEMM and the full GEMM with high hardware efficiency.
  • Figure 5: Decoder design for BSFP.
  • ...and 4 more figures