Cerberus: Efficient Inference with Adaptive Parallel Decoding and Sequential Knowledge Enhancement

TL;DR

Cerberus, an adaptive parallel decoding framework introduces the gating mechanism to enable the LLMs to adaptively choose appropriate decoding approaches at each decoding step, along with introducing a new paradigm of decoding heads that introduce the sequential knowledge while maintaining execution parallelism.

Abstract

Large language models (LLMs) often face a bottleneck in inference speed due to their reliance on auto-regressive decoding. Recently, parallel decoding has shown significant promise in enhancing inference efficiency. However, we have identified two key issues with existing parallel decoding frameworks: (1) decoding heads fail to balance prediction accuracy and the parallelism of execution, and (2) parallel decoding is not a universal solution, as it can bring unnecessary overheads at some challenging decoding steps. To address these issues, we propose Cerberus, an adaptive parallel decoding framework introduces the gating mechanism to enable the LLMs to adaptively choose appropriate decoding approaches at each decoding step, along with introducing a new paradigm of decoding heads that introduce the sequential knowledge while maintaining execution parallelism. The experiment results demonstrate that the Cerberus can achieve up to 2.12x speed up compared to auto-regressive decoding, and outperforms one of the leading parallel decoding frameworks, Medusa, with a 10% - 30% increase in acceleration and superior generation quality.

Figures (6)

• Figure 1: The implementation overview of Cerberus under two circumstances.
• Figure 2: The average inference latency per decoding step. Since Medusa requires additional operations compared to auto-regressive decoding, it will bring more time spent.
• Figure 3: The average entropy of the last hidden states when accepting different numbers of tokens, with setting 4 decoding heads. We conduct this experiment under three different tree paths, the setting of the tree path is a crucial component of tree verification, detailed presentation can be seen in Section \ref{['experimental setup']}.
• Figure 4: The architecture of Medusa's decoding heads (Medusa heads) and Cerberus's decoding heads (Cerberus heads). Each decoding head is composed of an FC layer and multiple Resblocks, the FC layer refers to a fully connected layer, and the detail of the Resblock is presented in Section \ref{['sec:cerberus head']} and Section \ref{['sec:gating mechanism']}.
• Figure 5: The comparison of top-$k$ accuracy between Medusa heads and Cerberus heads.