PROMISE: Process Reward Models Unlock Test-Time Scaling Laws in Generative Recommendations

TL;DR

This paper tackles Semantic Drift in SID-based Generative Recommendation caused by exposure bias, introducing Promise, a framework that combines a lightweight Path-level Process Reward Model (PRM) with PRM-guided Beam Search. The Path-level PRM is trained end-to-end with the generative backbone and uses dense, step-wise feedback (via an InfoNCE objective) to supervise intermediate SID paths, while the inference strategy prunes low-quality branches without increasing decoder compute. The approach yields Test-Time Scaling Laws, enabling smaller models to match or surpass larger ones by increasing inference compute, and it demonstrates strong offline and online performance gains on both public datasets and a large industrial platform (Kuaishou). The work provides extensive ablations, efficiency analyses, and a deployment blueprint, underscoring the practical value of dense intermediate supervision for scalable, high-quality generative recommendations.

Abstract

Generative Recommendation has emerged as a promising paradigm, reformulating recommendation as a sequence-to-sequence generation task over hierarchical Semantic IDs. However, existing methods suffer from a critical issue we term Semantic Drift, where errors in early, high-level tokens irreversibly divert the generation trajectory into irrelevant semantic subspaces. Inspired by Process Reward Models (PRMs) that enhance reasoning in Large Language Models, we propose Promise, a novel framework that integrates dense, step-by-step verification into generative models. Promise features a lightweight PRM to assess the quality of intermediate inference steps, coupled with a PRM-guided Beam Search strategy that leverages dense feedback to dynamically prune erroneous branches. Crucially, our approach unlocks Test-Time Scaling Laws for recommender systems: by increasing inference compute, smaller models can match or surpass larger models. Extensive offline experiments and online A/B tests on a large-scale platform demonstrate that Promise effectively mitigates Semantic Drift, significantly improving recommendation accuracy while enabling efficient deployment.

Figures (6)

• Figure 1: A specific case of SID generation based on the same user $u$ and context $c$ using standard beam search and PRM-guided beam search, respectively. (a): We observe the semantic drift where the model generates irrelevant SIDs, failing to retrieve target SIDs. (b): With PRM-guided search, the PRM evaluates and ranks the generation quality. The model successfully selects target SIDs and eliminates erroneous outputs caused by semantic drift.
• Figure 2: Overall framework of Promise. (a) and (b): An encoder-decoder architecture optimized via the NTP loss. (c): The Path-level PRM assesses the quality of SID reasoning paths. During training, an InfoNCE loss is used to maximize the score of the ground-truth SID path relative to negative paths. (d): An illustration of Model inference. At each generation step, the encoder-decoder produces a candidate set of $K^+$ SIDs. These are then evaluated by the Path-level PRM, which selects the top $K'<K^+$ optimal tokens to proceed with the next step of autoregressive prediction.
• Figure 3: Illustration of our test-time scaled beam search method compared with traditional beam search for SID generation.
• Figure 4: Test-time scaling laws can be validated by expanding $K^+$ in Promise. The red annotations indicate results of brute-force increasing the global beam size $K$, which significantly increases decoder computation. In contrast, our lightweight path-level PRM enables significant metric improvements through test-time scaling without adding decoder computation.
• Figure 5: Comparison of test-time scaling with parameter scaling.