Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding

Bin Xiao, Lujun Gui, Lei Su, Weipeng Chen

TL;DR

Clover-2 addresses GPU inefficiencies in autoregressive LLM decoding by advancing speculative decoding with a regressive speculator that leverages sequential information. It introduces four main enhancements—Information Extraction Order, Attention Decoder output projection, an Augmenting Block, and knowledge distillation—built on Clover to improve accuracy while maintaining low overhead. Empirical results on Vicuna-7B v1.5 and LLaMA3-Instruct-8B show Clover-2 achieves up to 3x throughput over vanilla decoding and outperforms both Clover and EAGLE in speculative token hit rate and speed, with notable gains across diverse tasks. The work demonstrates practical, deployable improvements for real-time serving of large language models, enabling faster and more reliable generation in production settings.

Abstract

Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.

Clover-2: Accurate Inference for Regressive Lightweight Speculative Decoding

TL;DR

Clover-2 addresses GPU inefficiencies in autoregressive LLM decoding by advancing speculative decoding with a regressive speculator that leverages sequential information. It introduces four main enhancements—Information Extraction Order, Attention Decoder output projection, an Augmenting Block, and knowledge distillation—built on Clover to improve accuracy while maintaining low overhead. Empirical results on Vicuna-7B v1.5 and LLaMA3-Instruct-8B show Clover-2 achieves up to 3x throughput over vanilla decoding and outperforms both Clover and EAGLE in speculative token hit rate and speed, with notable gains across diverse tasks. The work demonstrates practical, deployable improvements for real-time serving of large language models, enabling faster and more reliable generation in production settings.

Abstract

Large Language Models (LLMs) frequently suffer from inefficiencies, largely attributable to the discord between the requirements of auto-regressive decoding and the architecture of contemporary GPUs. Recently, regressive lightweight speculative decoding has garnered attention for its notable efficiency improvements in text generation tasks. This approach utilizes a lightweight regressive draft model, like a Recurrent Neural Network (RNN) or a single transformer decoder layer, leveraging sequential information to iteratively predict potential tokens. Specifically, RNN draft models are computationally economical but tend to deliver lower accuracy, while attention decoder layer models exhibit the opposite traits. This paper presents Clover-2, an advanced iteration of Clover, an RNN-based draft model designed to achieve comparable accuracy to that of attention decoder layer models while maintaining minimal computational overhead. Clover-2 enhances the model architecture and incorporates knowledge distillation to increase Clover's accuracy and improve overall efficiency. We conducted experiments using the open-source Vicuna 7B and LLaMA3-Instruct 8B models. The results demonstrate that Clover-2 surpasses existing methods across various model architectures, showcasing its efficacy and robustness.
Paper Structure (28 sections, 3 equations, 5 figures, 3 tables)

This paper contains 28 sections, 3 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Speculative Decoding and Tree Attention
  4. Clover Decoding
  5. EAGLE Decoding
  6. Clover-2
  7. Information extraction order
  8. Attention Decoder output projector
  9. Augmenting Block
  10. Knowledge distillation
  11. Other Details
  12. Evaluation
  13. Experiment Settings
  14. Models and baselines
  15. Dataset and Metrics
  16. ...and 13 more sections

Figures (5)

  • Figure 1: Overview of (a) Medusa, (b) EAGLE, (c) Clover, and (d) Clover-2.
  • Figure 2: A demonstration of Speculative Decoding and Tree Attention. Multiple speculations are merged by prefix matching to form a tree, and its topology dependency is represented in a 2-D matrix as the casual mask in Attention computation.
  • Figure 3: Detailed architecture design of Clover-2.
  • Figure 4: Number of extra generated tokens (excluding the first one) per step on various tasks.
  • Figure 5: The difference between compressed and regular tree mask structure