SAGE: Sequence-level Adaptive Gradient Evolution for Generative Recommendation

TL;DR

This work tackles generative recommendation by reusing open-source LLM architectures without a separate vocabulary, introducing SAGE (Sequence-level Adaptive Gradient Evolution) to overcome gradient-bound limitations. The approach combines data construction into natural language prompts, a two-stage training regime (SFT then RLHF), and an asymmetric, sequence-level optimization with multi-objective signal decoupling. SAGE employs a Positive Boost for cold-start items and an Entropy-Aware Penalty to prevent diversity collapse, while normalizing multi-objective signals to avoid reward collapse. Empirical results on Amazon benchmarks show RecLLM with SAGE outperforms prior baselines in accuracy, cold-start recovery, and diversity, highlighting the practical benefits of native-LLM reuse for scalable, diverse recommendation.

Abstract

While works such as OneRec have validated the scaling laws of Large Language Models (LLMs) in recommender systems, they rely on a cumbersome separate vocabulary. This dependency prevents the model architecture from reusing native LLM vocabularies, resulting in high maintenance costs and poor scalability. In response, we aim to efficiently reuse open-source LLM architectures without constructing a separate tokenization vocabulary. Furthermore, we identify that the optimization strategy of OneRec Gradient Bounded Policy Optimization (GBPO),suffers from a "Symmetric Conservatism" problem: its static gradient boundaries structurally suppress the update momentum required for cold-start items and fail to prevent diversity collapse in high-noise environments.To address this issue, we propose SAGE (Sequence-level Adaptive Gradient Evolution), a unified optimization framework tailored for list-wise generative recommendation. SAGE introduces two key innovations:(1) Sequence-level Signal Decoupling: By combining a geometric mean importance ratio with decoupled multi-objective advantages, we eliminate token-level variance and resolve the "Reward Collapse" problem. (2) Asymmetric Adaptive Dynamics: We construct a dynamic gradient manifold that applies a "Boost Factor" to high-potential cold start items to achieve super-linear updates and employs an "Entropy Aware Penalty" to break information cocoons. Theoretical analysis and empirical results demonstrate that SAGE effectively unblocks cold-start traffic and sustains recommendation diversity, all while retaining the numerical stability of GBPO.

Figures (5)

• Figure 1: Illustrates the motivation behind our research
• Figure 2: An overview of the modeling approach used in RecLLM-R1.
• Figure 3: Comparison of Gradient Boundaries. This figure illustrates the effective gradient coefficient ($r_{slate}/\Phi$) as a function of the importance ratio ($r_{slate}$). GBPO (Dashed Red) applies a static symmetric bound, capping positive updates at 1.0. In contrast, SAGE (Solid Blue) employs an asymmetric dynamic bound: it boosts high-potential positive samples beyond 1.0 (Case A) and applies entropy-aware penalties to negative samples (Case B), imposing stronger penalties on low-diversity slates (light blue) compared to high-diversity ones (dark blue).
• Figure 4: Evolution of Training Dynamics. (a) SAGE rapidly boosts the generation probability of cold-start items compared to GBPO, demonstrating the effectiveness of the Positive Boost mechanism. (b) SAGE maintains consistently higher list diversity throughout training, verifying that the Entropy-Aware Penalty successfully prevents mode collapse, whereas GBPO tends to converge towards lower-entropy states.
• Figure 5: Ablation Study on Amazon-Beauty Dataset. Metrics are normalized to the full SAGE model. Removing individual components leads to degradation in their respective targeted areas (e.g., removing Boost hurts Cold-Start most), validating the design of each module.