Efficient Inference for Large Language Model-based Generative Recommendation

TL;DR

This paper tackles the high latency of LLM-based generative recommenders by introducing AtSpeed, a speculative decoding framework designed for top-$K$ beam results under $N$-to-$K$ verification. It presents two strategies: AtSpeed-S for strict top-$K$ verification using RKLD-based alignment and density regularization, and AtSpeed-R for relaxed sampling verification using TVD-based alignment to allow high-probability non-top-$K$ drafts. A tree-based attention mechanism accelerates verification, and extensive experiments on real datasets show roughly 2x decoding speedups with minimal accuracy loss under relaxation, while maintaining exact top-$K$ results under strict verification. The work demonstrates practical, lossless acceleration for LLM-based recommendations and provides reproducible code and datasets, enabling broader deployment of efficient generative recommender systems.

Abstract

Large Language Model (LLM)-based generative recommendation has achieved notable success, yet its practical deployment is costly particularly due to excessive inference latency caused by autoregressive decoding. For lossless LLM decoding acceleration, Speculative Decoding (SD) has emerged as a promising solution. However, applying SD to generative recommendation presents unique challenges due to the requirement of generating top-K items (i.e., K distinct token sequences) as a recommendation list by beam search. This leads to more stringent verification in SD, where all the top-K sequences from the target LLM must be successfully drafted by the draft model at each decoding step. To alleviate this, we consider 1) boosting top-K sequence alignment between the draft model and the target LLM, and 2) relaxing the verification strategy to reduce trivial LLM calls. To this end, we propose an alignment framework named AtSpeed, which presents the AtSpeed-S optimization objective for top-K alignment under the strict top-K verification. Moreover, we introduce a relaxed sampling verification strategy that allows high-probability non-top-K drafted sequences to be accepted, significantly reducing LLM calls. Correspondingly, we propose AtSpeed-R for top-K alignment under this relaxed sampling verification. Empirical results on two real-world datasets demonstrate that AtSpeed significantly accelerates LLM-based generative recommendation, e.g., near 2x speedup under strict top-K verification and up to 2.5x speedup under relaxed sampling verification. The codes and datasets are released at https://github.com/Linxyhaha/AtSpeed.

Figures (7)

• Figure 1: (a) The inference time costs of LC-Rec zheng2023adapting with LLaMA-7B on a single A5000 GPU. (b) The illustration of the $N$-to-1 verification of SD with greedy decoding in NLP tasks. (c) $N$-to-$K$ verification of SD with beam search in recommendation tasks, where the drafting length, candidate number $N$, and beam size $K$ are set at 3 for illustration.
• Figure 2: Overview of AtSpeed. (a) shows the alignment training of the draft model with an additional alignment loss tailored for different verification strategies, e.g., $\mathcal{L}_{\text{Align-R}}$. (b) depicts the AtSpeed inference, where the well-trained draft model produces beam search sequences from each step, i.e., ${\mathcal{Y}_{j}^{q}}$, for the target LLM to verify. The beam sequences from the last accepted step before encountering the first rejection are utilized in the following SD step, i.e., $\mathcal{Y}_{2}^q$. (c) demonstrates the tree-based attention for the drafted beam sequences ($N$=2 and $\gamma$=3).
• Figure 3: Ablation study of AtSpeed-S and AtSpeed-R on Beauty dataset.
• Figure 4: Effect of draft beam size $N$ and the alignment strength $\alpha$ on Beauty dataset.
• Figure 5: (a) Speedup effect of tree-based attention on Beauty. (b) Performance of AtSpeed on varying hyper-parameters (alignment strength $\alpha$) on Beauty.

Theorems & Definitions (7)

• Theorem 1
• proof
• Definition 1
• Lemma 1
• proof
• Theorem 2
• proof