Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution

Siqi Wang, Hailong Yang, Xuezhu Wang, Tongxuan Liu, Pengbo Wang, Xuning Liang, Kejie Ma, Tianyu Feng, Xin You, Yongjun Bao, Yi Liu, Zhongzhi Luan, Depei Qian

TL;DR

This paper proposes Minions, an LLM inference system that accelerates LLM inference with a collective and adaptive speculative generation and proposes an adaptive mechanism to dynamically determine the optimal speculation length of SSM, which can achieve better inference performance across different models, datasets, and hyper-parameters.

Abstract

Large language models (LLM) have recently attracted surging interest due to their outstanding capabilities across various domains. However, enabling efficient LLM inference is challenging due to its autoregressive decoding that generates tokens only one at a time. Although research works apply pruning or quantization to speed up LLM inference, they typically require fine-tuning the LLM, incurring significant time and economic costs. Meanwhile, speculative decoding has been proposed to use small speculative models (SSMs) to accelerate the inference of LLM. However, the low acceptance rate of SSM and the high verification cost of LLM prohibit further performance improvement of inference. In this paper, we propose Minions, an LLM inference system that accelerates LLM inference with a collective and adaptive speculative generation. Specifically, Minions proposes a majority-voted mechanism to leverage multiple SSMs to jointly speculate the outputs of LLM, which improves the inference performance without introducing prohibitive computation costs for LLM. To better trade off the number of tokens speculated from SSM and the verification cost of LLM, Minions proposes an adaptive mechanism to dynamically determine the optimal speculation length of SSM, which can achieve better inference performance across different models, datasets, and hyper-parameters. In addition, Minions decouples the SSM decoding and LLM verification efficiently and adopts a pipelined execution mechanism to further improve the inference performance of LLM. By comparing with the state-of-the-art LLM inference systems, we demonstrate that Minions can achieve higher inference throughput and lower inference time.

Minions: Accelerating Large Language Model Inference with Aggregated Speculative Execution

TL;DR

This paper proposes Minions, an LLM inference system that accelerates LLM inference with a collective and adaptive speculative generation and proposes an adaptive mechanism to dynamically determine the optimal speculation length of SSM, which can achieve better inference performance across different models, datasets, and hyper-parameters.

Abstract

Large language models (LLM) have recently attracted surging interest due to their outstanding capabilities across various domains. However, enabling efficient LLM inference is challenging due to its autoregressive decoding that generates tokens only one at a time. Although research works apply pruning or quantization to speed up LLM inference, they typically require fine-tuning the LLM, incurring significant time and economic costs. Meanwhile, speculative decoding has been proposed to use small speculative models (SSMs) to accelerate the inference of LLM. However, the low acceptance rate of SSM and the high verification cost of LLM prohibit further performance improvement of inference. In this paper, we propose Minions, an LLM inference system that accelerates LLM inference with a collective and adaptive speculative generation. Specifically, Minions proposes a majority-voted mechanism to leverage multiple SSMs to jointly speculate the outputs of LLM, which improves the inference performance without introducing prohibitive computation costs for LLM. To better trade off the number of tokens speculated from SSM and the verification cost of LLM, Minions proposes an adaptive mechanism to dynamically determine the optimal speculation length of SSM, which can achieve better inference performance across different models, datasets, and hyper-parameters. In addition, Minions decouples the SSM decoding and LLM verification efficiently and adopts a pipelined execution mechanism to further improve the inference performance of LLM. By comparing with the state-of-the-art LLM inference systems, we demonstrate that Minions can achieve higher inference throughput and lower inference time.
Paper Structure (21 sections, 4 equations, 14 figures, 1 table)

This paper contains 21 sections, 4 equations, 14 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Speculative Decoding
  4. Model Capability Gap Between SSM and LLM
  5. Vast Search Space for Determining Optimal Speculation Length
  6. Tightly-Coupled Speculative Execution Hinders Inference Performance
  7. Methodology
  8. Design Overview
  9. Majority-voted Speculator
  10. Adaptive Speculation Length Selector
  11. Speculative Generation Pipeline
  12. Implementation
  13. Evaluation
  14. Experiment Setup
  15. End-to-end Performance
  16. ...and 6 more sections

Figures (14)

  • Figure 1: The acceptance rate of models with different speculation lengths and datasets.
  • Figure 2: The inference time of the LLM under different speculation lengths, datasets, and batch size.
  • Figure 3: Performance potential revealed by pipelined execution. $T$ represents the reduced inference time.
  • Figure 4: Design overview of Minions
  • Figure 5: The primary workflow of Majority-voted Speculator.
  • ...and 9 more figures