Towards Sample-Efficient and Stable Reinforcement Learning for LLM-based Recommendation

TL;DR

This work argues that Long CoT reasoning is ill-suited for sequential recommendation due to high inference latency and mismatches in cognitive signals. It shifts focus to reinforcement learning, introducing RISER to perform end-to-end item-space exploration with a two-stage SFT+RL pipeline. RISER combines SimPO for learning from failed rollouts, oversampling with de-duplication, a certainty-aware token mask, and KL-Cov-based entropy control to achieve stable, sample-efficient learning. Extensive experiments on three real-world datasets show RISER consistently outperforms strong baselines, including reasoning-based methods, validating its effectiveness and robustness in RL-enhanced LLM-based recommendations. The approach offers a practical, scalable paradigm for deploying RL in large language model–driven recommender systems.

Abstract

While Long Chain-of-Thought (Long CoT) reasoning has shown promise in Large Language Models (LLMs), its adoption for enhancing recommendation quality is growing rapidly. In this work, we critically examine this trend and argue that Long CoT is inherently ill-suited for the sequential recommendation domain. We attribute this misalignment to two primary factors: excessive inference latency and the lack of explicit cognitive reasoning patterns in user behavioral data. Driven by these observations, we propose pivoting away from the CoT structure to directly leverage its underlying mechanism: Reinforcement Learning (RL), to explore the item space. However, applying RL directly faces significant obstacles, notably low sample efficiency-where most actions fail to provide learning signals-and training instability. To overcome these limitations, we propose RISER, a novel Reinforced Item Space Exploration framework for Recommendation. RISER is designed to transform non-learnable trajectories into effective pairwise preference data for optimization. Furthermore, it incorporates specific strategies to ensure stability, including the prevention of redundant rollouts and the constraint of token-level update magnitudes. Extensive experiments on three real-world datasets show that RISER significantly outperforms competitive baselines, establishing a robust paradigm for RL-enhanced LLM recommendation. Our code will be available at https://anonymous.4open.science/r/RISER/.

Figures (3)

• Figure 1: Overview of the RISER framework. The final loss, $\mathcal{L}_{\text{RISER}}$ is a weighted sum of two objectives applied to different data partitions: a modified GRPO objective for successful rollouts and a SimPO objective for failed rollouts. Each core mechanism is detailed in the corresponding sub-diagram.
• Figure 2: Comparison of our method and SFT-trained method over popular and unpopular item groups on Games dataset, reporting NDCG@10 (Left) and HR@10 (Right).
• Figure 5: Comparison of policy entropy and test set HR@1 for RISER and the SFT+RL baseline over the initial 400 training steps on the Toys and Goodreads datasets.