Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PLD+: Accelerating LLM inference by leveraging Language Model Artifacts

Shwetha Somasundaram, Anirudh Phukan, Apoorv Saxena

TL;DR

PLD+ introduces a tuning-free, plug-and-play speculative decoding framework that accelerates LLM inference by drafting tokens from input overlaps and ranking them with model artifacts (attentions or hidden states). It avoids extra training and draft-models, delivering substantial speedups across multiple input-guided tasks and model families, sometimes surpassing tuning-dependent SOTA methods. The approach relies on selective attention-head targeting and hidden-state similarities to identify optimal drafts, with extensive ablations demonstrating robust performance and practical applicability. Overall, PLD+ offers a practical, model-agnostic method to speed up inference for real-world, input-rich tasks without incurring additional compute or tuning costs.

Abstract

To reduce the latency associated with autoretrogressive LLM inference, speculative decoding has emerged as a novel decoding paradigm, where future tokens are drafted and verified in parallel. However, the practical deployment of speculative decoding is hindered by its requirements for additional computational resources and fine-tuning, which limits its out-of-the-box usability. To address these challenges, we present PLD+, a suite of novel algorithms developed to accelerate the inference process of LLMs, particularly for input-guided tasks. These tasks, which include code editing, text editing, summarization, etc., often feature outputs with substantial overlap with their inputs-an attribute PLD+ is designed to exploit. PLD+ also leverages the artifacts (attention and hidden states) generated during inference to accelerate inference speed. We test our approach on five input-guided tasks and through extensive experiments we find that PLD+ outperforms all tuning-free approaches. In the greedy setting, it even outperforms the state-of-the-art tuning-dependent approach EAGLE on four of the tasks. (by a margin of upto 2.31 in terms of avg. speedup). Our approach is tuning free, does not require any additional compute and can easily be used for accelerating inference of any LLM.

PLD+: Accelerating LLM inference by leveraging Language Model Artifacts

TL;DR

PLD+ introduces a tuning-free, plug-and-play speculative decoding framework that accelerates LLM inference by drafting tokens from input overlaps and ranking them with model artifacts (attentions or hidden states). It avoids extra training and draft-models, delivering substantial speedups across multiple input-guided tasks and model families, sometimes surpassing tuning-dependent SOTA methods. The approach relies on selective attention-head targeting and hidden-state similarities to identify optimal drafts, with extensive ablations demonstrating robust performance and practical applicability. Overall, PLD+ offers a practical, model-agnostic method to speed up inference for real-world, input-rich tasks without incurring additional compute or tuning costs.

Abstract

To reduce the latency associated with autoretrogressive LLM inference, speculative decoding has emerged as a novel decoding paradigm, where future tokens are drafted and verified in parallel. However, the practical deployment of speculative decoding is hindered by its requirements for additional computational resources and fine-tuning, which limits its out-of-the-box usability. To address these challenges, we present PLD+, a suite of novel algorithms developed to accelerate the inference process of LLMs, particularly for input-guided tasks. These tasks, which include code editing, text editing, summarization, etc., often feature outputs with substantial overlap with their inputs-an attribute PLD+ is designed to exploit. PLD+ also leverages the artifacts (attention and hidden states) generated during inference to accelerate inference speed. We test our approach on five input-guided tasks and through extensive experiments we find that PLD+ outperforms all tuning-free approaches. In the greedy setting, it even outperforms the state-of-the-art tuning-dependent approach EAGLE on four of the tasks. (by a margin of upto 2.31 in terms of avg. speedup). Our approach is tuning free, does not require any additional compute and can easily be used for accelerating inference of any LLM.

Paper Structure

This paper contains 38 sections, 4 equations, 5 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Background: Autoregressive decoding and Speculative Decoding
  4. Our approach: PLD+
  5. Notations
  6. Drafting Algorithm
  7. Ranking occurrences using model attentions
  8. Ranking occurrences using hidden states
  9. Draft Prediction
  10. Verification Algorithm
  11. Application Scenarios
  12. Experimental Setup
  13. Datasets
  14. Baselines
  15. Metrics
  16. ...and 23 more sections

Figures (5)

  • Figure 1: Overview of PLD+. During the generation of token $t$, possible drafts are retrieved from the context by searching for the same tokens as token $t-1$ ("to"). These candidates are then ranked using the information present in the model artifacts (hidden states in the figure) corresponding to token $t-2$. The text following "to" in the highest ranked candidate is proposed as the draft. The draft is verified against the actual LLM generation and all successfully verified tokens are generated in time step $t$, resulting in speedup.
  • Figure 2: As shown in the figure, the last generated token is \n . PLD has proposed draft tokens by selecting the candidate span with the longest matching prefix. However this is not always the optimum choice as shown in the figure. PLD+ proposes the candidate span with the highest semantic relevance resulting draft token acceptance.
  • Figure 3: In this figure, we display the results of aggregating attentions across the heads of a specific layer. We experiment with two aggregation operations - summation and maximum aggregation.
  • Figure 4: Effect of K, number of draft tokens on the performance of PLD, PLD+
  • Figure 5: Effect of layer $l$ on performance of PLD+ (h)