Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning

TL;DR

This paper tackles the challenges of training LLM-based conversational recommender systems by introducing ConvRec-R1, a two-stage framework combining Remap-Reflect-Adjust for catalog-grounded supervised fine-tuning and Rank-GRPO for rank-aware reinforcement learning. Rank-GRPO reformulates policy updates to treat each rank in the recommendation list as the unit of action, using rank-level advantages and a geometric-mean-based probability to stabilize learning and address non-causal credit assignment. Empirical results on Reddit-v2 show faster convergence and superior Recall and NDCG compared to GRPO-type baselines, with open-source backbones matching or surpassing larger black-box models in many settings. The approach demonstrates strong grounding and ranking performance and offers a pathway to applying rank-structured RL to other retrieval and ranking tasks beyond conversational recommendations.

Abstract

Large language models (LLMs) are reshaping the recommender system paradigm by enabling users to express preferences and receive recommendations through conversations. Yet, aligning LLMs to the recommendation task remains challenging: pretrained LLMs often generate out-of-catalog items, violate required output formats, and their ranking quality degrades sharply toward the end of the generated list. To this end, we propose ConvRec-R1, a two-stage framework for end-to-end training of LLM-based conversational recommender systems. In Stage 1, we construct a behavioral-cloning dataset with a Remap-Reflect-Adjust pipeline, which produces high-quality, catalog-grounded demonstrations from powerful blackbox LLMs to warm-start the RL training. In Stage 2, we propose Rank-GRPO, a principled extension of group relative policy optimization (GRPO) tailored to tasks with rank-style outputs. Rank-GRPO treats each rank in the recommendation list as the unit instead of token (too fine-grained) or sequence (too coarse), redefining rewards to remove non-causal credit assignment and introducing a rank-level importance ratio based on the geometric mean of rank-wise token probabilities to stabilize policy updates. Experiments on the public Reddit-v2 dataset show that ConvRec-R1 converges faster and achieves higher Recall and NDCG than GRPO-style baselines. Code and datasets are released at https://github.com/yaochenzhu/Rank-GRPO.

Figures (10)

• Figure 1: Training dynamics on loss, validation metrics, validation in-catalg recommendation ratio with different backbone models during the SFT stage on the Reddit-v2 dataset.
• Figure 2: Left + Middle: Training dynamics of the reward acquired by ConvRec-R1 with Rank-GRPO on different rank during the RL stage. Right: Comparison of validation NDCG between GRPO and Rank-GRPO (both on-policy) on the Reddit-v2 dataset.
• Figure 3: Training dynamic of reward and validation NDCG between GRPO, GSPO and Rank-GRPO (off-policy) on Reddit-v2. See Appendix Fig. \ref{['fig:llama_1b+rl_off_reward']} for results with Llama3.2-1B-Instruct.
• Figure 4: Training dynamics of ConvRec-R1 with Llama3.2-1B-Instruct backbone (on-policy).
• Figure 5: Comparison of training dynamics of reward and validation NDCG between GRPO, GSPO and Rank-GRPO (off-policy) with Llama3.2-1B-Instruct backbone on the Reddit-v2 dataset.