Lossless Acceleration of Large Language Model via Adaptive N-gram Parallel Decoding

TL;DR

The paper tackles the latency of autoregressive LLMs by introducing Adaptive N-gram Parallel Decoding (ANPD), a lossless, plug-and-play acceleration that drafts multiple tokens in parallel using an adaptive N-gram module and then verifies them with the original LLM. It uses a two-stage process and a Multi-Level N-gram (MLN) to improve draft quality without retraining or extra draft models, preserving output fidelity. Experiments across LLaMA, Alpaca, CodeLLaMA, and ChatGLM3 on summarization and code generation show speed-ups up to 3.67x, with robust performance across models and tasks. The approach enables practical deployment of large language models in latency-sensitive settings by reducing inference time without compromising output integrity.

Abstract

While Large Language Models (LLMs) have shown remarkable abilities, they are hindered by significant resource consumption and considerable latency due to autoregressive processing. In this study, we introduce Adaptive N-gram Parallel Decoding (ANPD), an innovative and lossless approach that accelerates inference by allowing the simultaneous generation of multiple tokens. ANPD incorporates a two-stage approach: it begins with a rapid drafting phase that employs an N-gram module, which adapts based on the current interactive context, followed by a verification phase, during which the original LLM assesses and confirms the proposed tokens. Consequently, ANPD preserves the integrity of the LLM's original output while enhancing processing speed. We further leverage a multi-level architecture for the N-gram module to enhance the precision of the initial draft, consequently reducing inference latency. ANPD eliminates the need for retraining or extra GPU memory, making it an efficient and plug-and-play enhancement. In our experiments, models such as LLaMA and its fine-tuned variants have shown speed improvements up to 3.67x, validating the effectiveness of our proposed ANPD.

Figures (6)

• Figure 1: The comparative analysis of the number of words and tokens after tokenizer processing for the CNN/Daily Mail and XSUM datasets.
• Figure 2: The pipeline of the ANPD. The tokenizer first processes the text to obtain a list of tokens. These tokens are used to initialize the N-gram module. Simultaneously, these tokens are fed into the LLM for processing via autoregression. The predicted token at time $t_0$ in the figure is "_Very". This word is used as a query into the N-gram module, yielding the token "_Re", which along with the "_Very" are sent to the LLM for inference at time $t_1$. A green checkmark signifies acceptance of the predicted token, while a red cross indicates rejection. Each accepted token, is combined with the first $N-1$ tokens to form a tuple, and the update method is called to refresh the N-gram module.
• Figure 3: Speed up ratio of LLM for different $K$.
• Figure 4: Decoding speed up ratio of LLaMA-7B for different $K$ and $N$.
• Figure 5: Decoding speed up ratio of Alpaca-CNN/DM-7B for different $K$ and $N$.