Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting

Jiaming Xu, Jiayi Pan, Yongkang Zhou, Siming Chen, Jinhao Li, Yaoxiu Lian, Junyi Wu, Guohao Dai

TL;DR

SpecEE addresses the latency and cost of large language model inference by introducing speculative early exiting. It combines a speculation-based lightweight predictor, a two-level scheduling engine, and a context-aware merged mapping to enable speculative decoding with minimal training overhead. Across cloud and PC deployments, SpecEE achieves substantial end-to-end speedups while preserving accuracy and remains compatible with orthogonal acceleration techniques such as quantization and sparse activation. The work demonstrates a practical pathway to push the Pareto frontier of speed and accuracy for real-world LLM inference.

Abstract

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access. In this paper, we present SpecEE, a fast LLM inference engine with speculative early exiting. (1) At the algorithm level, we propose the speculation-based lightweight predictor design by exploiting the probabilistic correlation between the speculative tokens and the correct results and high parallelism of GPUs. (2) At the system level, we point out that not all layers need a predictor and design the two-level heuristic predictor scheduling engine based on skewed distribution and contextual similarity. (3) At the mapping level, we point out that different decoding methods share the same essential characteristics, and propose the context-aware merged mapping for predictor with efficient GPU implementations to support speculative decoding, and form a framework for various existing orthogonal acceleration techniques (e.g., quantization and sparse activation) on cloud and personal computer (PC) scenarios, successfully pushing the Pareto frontier of accuracy and speedup. It is worth noting that SpecEE can be applied to any LLM by negligible training overhead in advance without affecting the model original parameters. Extensive experiments show that SpecEE achieves 2.25x and 2.43x speedup with Llama2-7B on cloud and PC scenarios respectively.

SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting

TL;DR

SpecEE addresses the latency and cost of large language model inference by introducing speculative early exiting. It combines a speculation-based lightweight predictor, a two-level scheduling engine, and a context-aware merged mapping to enable speculative decoding with minimal training overhead. Across cloud and PC deployments, SpecEE achieves substantial end-to-end speedups while preserving accuracy and remains compatible with orthogonal acceleration techniques such as quantization and sparse activation. The work demonstrates a practical pathway to push the Pareto frontier of speed and accuracy for real-world LLM inference.

Abstract

Early exiting has recently emerged as a promising technique for accelerating large language models (LLMs) by effectively reducing the hardware computation and memory access. In this paper, we present SpecEE, a fast LLM inference engine with speculative early exiting. (1) At the algorithm level, we propose the speculation-based lightweight predictor design by exploiting the probabilistic correlation between the speculative tokens and the correct results and high parallelism of GPUs. (2) At the system level, we point out that not all layers need a predictor and design the two-level heuristic predictor scheduling engine based on skewed distribution and contextual similarity. (3) At the mapping level, we point out that different decoding methods share the same essential characteristics, and propose the context-aware merged mapping for predictor with efficient GPU implementations to support speculative decoding, and form a framework for various existing orthogonal acceleration techniques (e.g., quantization and sparse activation) on cloud and personal computer (PC) scenarios, successfully pushing the Pareto frontier of accuracy and speedup. It is worth noting that SpecEE can be applied to any LLM by negligible training overhead in advance without affecting the model original parameters. Extensive experiments show that SpecEE achieves 2.25x and 2.43x speedup with Llama2-7B on cloud and PC scenarios respectively.

Paper Structure

This paper contains 45 sections, 19 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Large Language Models
  4. Speculative Decoding in LLMs
  5. Skip Layer and Early Exiting
  6. Motivation
  7. Key Challenges of Early Exiting
  8. Key Insight
  9. Speculation-based Lightweight Predictor
  10. Motivation: Time-consuming Predictor
  11. Insight: Probability Shift
  12. Approach: Lightweight Design
  13. Feature Selection.
  14. Judgment Mechanism.
  15. Verification Algorithm.
  16. ...and 30 more sections

Figures (19)

  • Figure 1: (a) Pareto frontier of accuracy and speedup towards LLM inference and deployment. The detailed normalized accuracy and speedup are obtained with Llama2-7B on an NVIDIA RTX 4090 GPU. (b) The ratio of the time of the decoder layer to end-to-end inference time in the original LLM. The data of two decodings is obtained based on Hugging Face huggingface and EAGLE li2024eagle frameworks. (c) Different numbers of decoder layers are needed for different token generation.
  • Figure 2: Overview of SpecEE. (a) Dataflow of early exiting. (b) Key insight: Speculative model can reduce vocabulary space for predictor. (c) Techniques on predictor optimization from Section \ref{['sec:T1']} to Section \ref{['sec:T3']}. (d) Results on cloud and PC scenarios.
  • Figure 3: Architecture of SpecEE.
  • Figure 4: Two decoding methods of LLM.
  • Figure 5: (a) The insight on probability shift detailed in Section \ref{['sec:T1:analysis']}. (b) The algorithm flow chart and the feature extraction in speculation-based vocabulary space reduction.
  • ...and 14 more figures