Reinforced Preference Optimization for Recommendation

TL;DR

ReVerges on a reinforcement-based paradigm for generative recommenders by introducing Reinforced Preference Optimization for Recommendation (ReRe), which combines constrained decoding, on-policy beam-search sampling, and a unified reward that blends rule-based accuracy with ranking signals. By addressing the negative sampling weakness and reward sparsity of prior DL/RL approaches, ReRe achieves substantial ranking gains across multiple real-world datasets and backbone models, with robust generalization across model families and scales. The work systematically analyzes generation, sampling, reward design, and optimization choices, showing that on-policy, diversity-preserving sampling paired with a ranking-aware reward yields the strongest alignment with true user preferences. This approach offers a practical framework for applying reinforcement learning to LLM-based recommender systems and provides actionable insights for future RL-based recommendation research.

Abstract

Recent breakthroughs in large language models (LLMs) have fundamentally shifted recommender systems from discriminative to generative paradigms, where user behavior modeling is achieved by generating target items conditioned on historical interactions. Yet current generative recommenders still suffer from two core limitations: the lack of high-quality negative modeling and the reliance on implicit rewards. Reinforcement learning with verifiable rewards (RLVR) offers a natural solution by enabling on-policy sampling of harder negatives and grounding optimization in explicit reward signals. However, applying RLVR to generative recommenders remains non-trivial. Its unique generation space often leads to invalid or repetitive items that undermine sampling efficiency, and ranking supervision is sparse since most items receive identical zero rewards. To address these challenges, we propose Reinforced Preference Optimization for Recommendation (ReRe), a reinforcement-based paradigm tailored to LLM-based recommenders, an important direction in generative recommendation. ReRe incorporates constrained beam search to improve sampling efficiency and diversify hard negatives, while augmenting rule-based accuracy rewards with auxiliary ranking rewards for finer-grained supervision. Extensive experiments on three real-world datasets demonstrate that ReRe consistently outperforms both traditional and LLM-based recommenders in ranking performance. Further analysis shows that ReRe not only enhances performance across both base and SFT-initialized models but also generalizes robustly across different backbone families and scales. Beyond empirical gains, we systematically investigate the design space of RLVR in recommendation across generation, sampling strategy, reward modeling, and optimization algorithm, offering insights for future research.

Figures (6)

• Figure 1: Comparison of negative item source across different post-training methods. Notably, DPO samples random negative items while self-play DPO samples negative items from the frozen reference model $\pi_\text{ref}$. In contrast, RLVR directly samples negative items from the model $\pi_\theta$ being updated.
• Figure 2: Framework of ReRe, where gt.means ground truth and $ra_k=-\frac{1}{\log(1+k)}$ . At each iteration, ReRe first samples $G$ different items from LLM-based recommenders given the prompt containing user interactions. The generated items are then assigned the rule-based reward plus the ranking reward, from which the advantages are computed to update the model via GRPO.
• Figure 3: Comparison among the consistency of three rewards and NDCG@10 metrics on Industrial.
• Figure 4: Study on Industrial: (\ref{['fig:sdpo-alignment']}) Reward margin and N@10 during S-DPO training. (\ref{['fig:indusrial_diversity']}) Diversity evolution of different sampling strategies. (\ref{['fig:indusrial_G']}) Effect of the number of generated items $G$ in ReRe.
• Figure 5: (\ref{['fig:diversity_toys']}) Divsersity evolution of different sampling strategies on Toys. (\ref{['fig:sdpo_toys']}) Reward margin and N@10 during S-DPO training on Toys. (\ref{['fig:industrial_beta']}) The influence of the value of $\beta$ on ReRe (Industrial).