Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models

Mahsa Khoshnoodi, Vinija Jain, Mingye Gao, Malavika Srikanth, Aman Chadha

TL;DR

This survey tackles the latency challenge of autoregressive LLM inference by organizing accelerated generation methods into three main families: speculative decoding, early exiting, and non-autoregressive decoding. It systematically analyzes drafting and verification in speculative decoding, dynamic computation and confidence-based exits, and parallel decoding in non-autoregressive models, surveying a wide range of techniques and reported speedups. The work provides a taxonomy, compares underlying principles, discusses practical trade-offs (accuracy, memory, integration), and highlights future directions for robust deployment in real-world settings. By synthesizing methods across architectures and tasks, the paper offers a roadmap for researchers and practitioners seeking faster, scalable LLM inference without sacrificing quality.

Abstract

Despite the crucial importance of accelerating text generation in large language models (LLMs) for efficiently producing content, the sequential nature of this process often leads to high inference latency, posing challenges for real-time applications. Various techniques have been proposed and developed to address these challenges and improve efficiency. This paper presents a comprehensive survey of accelerated generation techniques in autoregressive language models, aiming to understand the state-of-the-art methods and their applications. We categorize these techniques into several key areas: speculative decoding, early exiting mechanisms, and non-autoregressive methods. We discuss each category's underlying principles, advantages, limitations, and recent advancements. Through this survey, we aim to offer insights into the current landscape of techniques in LLMs and provide guidance for future research directions in this critical area of natural language processing.

A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models

TL;DR

This survey tackles the latency challenge of autoregressive LLM inference by organizing accelerated generation methods into three main families: speculative decoding, early exiting, and non-autoregressive decoding. It systematically analyzes drafting and verification in speculative decoding, dynamic computation and confidence-based exits, and parallel decoding in non-autoregressive models, surveying a wide range of techniques and reported speedups. The work provides a taxonomy, compares underlying principles, discusses practical trade-offs (accuracy, memory, integration), and highlights future directions for robust deployment in real-world settings. By synthesizing methods across architectures and tasks, the paper offers a roadmap for researchers and practitioners seeking faster, scalable LLM inference without sacrificing quality.

Abstract

Despite the crucial importance of accelerating text generation in large language models (LLMs) for efficiently producing content, the sequential nature of this process often leads to high inference latency, posing challenges for real-time applications. Various techniques have been proposed and developed to address these challenges and improve efficiency. This paper presents a comprehensive survey of accelerated generation techniques in autoregressive language models, aiming to understand the state-of-the-art methods and their applications. We categorize these techniques into several key areas: speculative decoding, early exiting mechanisms, and non-autoregressive methods. We discuss each category's underlying principles, advantages, limitations, and recent advancements. Through this survey, we aim to offer insights into the current landscape of techniques in LLMs and provide guidance for future research directions in this critical area of natural language processing.
Paper Structure (9 sections, 4 equations, 2 figures)

This paper contains 9 sections, 4 equations, 2 figures.

Table of Contents

  1. Introduction
  2. Speculative Decoding
  3. Methods Targeting Drafting Phase
  4. Methods Targeting Verification Phase
  5. Integrated Methods for Enhanced Speculative Decoding
  6. Early Exiting
  7. Non-autoregressive Models
  8. Discussion and Limitations
  9. Conclusion

Figures (2)

  • Figure 1: Taxonomy of accelerated generation techniques in LLMs is categorized into Speculative Decoding, Early exiting, and Non-Autoregressive methods. Each category presents unique strategies aimed at optimizing model efficiency and efficacy. Speculative decoding explores multiple candidate outputs simultaneously, while early exiting prioritizes termination upon confident predictions. Non-autoregressive methods introduce innovative approaches to parallelization and coherent output generation.
  • Figure 2: The three core steps of the blockwise algorithm