Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Yijun Pan, Weikang Qiu, Qiyao Ma, Mingxuan Ju, Tong Zhao, Neil Shah, Rex Ying

Abstract

Modern recommender systems must adapt to dynamic, need-specific objectives for diverse recommendation scenarios, yet most traditional recommenders are optimized for a single static target and struggle to reconfigure behavior on demand. Recent advances in reinforcement-learning-based post-training have unlocked strong instruction-following and reasoning capabilities in LLMs, suggesting a principled route for aligning them to complex recommendation goals. Motivated by this, we study closed-set autoregressive ranking, where an LLM generates a permutation over a fixed candidate set conditioned on user context and an explicit need instruction. However, applying RL to this setting faces two key obstacles: (i) sequence-level rewards yield coarse credit assignment that fails to provide fine-grained training signals, and (ii) interaction feedback is sparse and noisy, which together lead to inefficient and unstable updates. We propose FlexRec, a post-training RL framework that addresses both issues with (1) a causally grounded item-level reward based on counterfactual swaps within the remaining candidate pool, and (2) critic-guided, uncertainty-aware scaling that explicitly models reward uncertainty and down-weights low-confidence rewards to stabilize learning under sparse supervision. Across diverse recommendation scenarios and objectives, FlexRec achieves substantial gains: it improves NDCG@5 by up to \textbf{59\%} and Recall@5 by up to \textbf{109.4\%} in need-specific ranking, and further achieves up to \textbf{24.1\%} Recall@5 improvement under generalization settings, outperforming strong traditional recommenders and LLM-based baselines.

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Abstract

Modern recommender systems must adapt to dynamic, need-specific objectives for diverse recommendation scenarios, yet most traditional recommenders are optimized for a single static target and struggle to reconfigure behavior on demand. Recent advances in reinforcement-learning-based post-training have unlocked strong instruction-following and reasoning capabilities in LLMs, suggesting a principled route for aligning them to complex recommendation goals. Motivated by this, we study closed-set autoregressive ranking, where an LLM generates a permutation over a fixed candidate set conditioned on user context and an explicit need instruction. However, applying RL to this setting faces two key obstacles: (i) sequence-level rewards yield coarse credit assignment that fails to provide fine-grained training signals, and (ii) interaction feedback is sparse and noisy, which together lead to inefficient and unstable updates. We propose FlexRec, a post-training RL framework that addresses both issues with (1) a causally grounded item-level reward based on counterfactual swaps within the remaining candidate pool, and (2) critic-guided, uncertainty-aware scaling that explicitly models reward uncertainty and down-weights low-confidence rewards to stabilize learning under sparse supervision. Across diverse recommendation scenarios and objectives, FlexRec achieves substantial gains: it improves NDCG@5 by up to \textbf{59\%} and Recall@5 by up to \textbf{109.4\%} in need-specific ranking, and further achieves up to \textbf{24.1\%} Recall@5 improvement under generalization settings, outperforming strong traditional recommenders and LLM-based baselines.
Paper Structure (65 sections, 18 equations, 4 figures, 9 tables)

This paper contains 65 sections, 18 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLMs for Multi-Need Recommendation
  4. Reinforcement Learning from Verifiable Rewards
  5. Sparse User Interactions
  6. Method
  7. Problem Formulation: Need-specific Autoregressive Ranking
  8. Swap-based Item-level Reward
  9. Properties of Autoregressive Reranking.
  10. Reward Design
  11. Time Complexity
  12. Item-Level GRPO
  13. Sequence-Level Advantages for Non-Item Tokens
  14. Uncertainty-Aware GRPO
  15. Uncertainty of Sequence Reward
  16. ...and 50 more sections

Figures (4)

  • Figure 1: Overall framework of FlexRec. Given a candidate set and an explicit user need, an LLM recommender generates ranked rollouts. An item-level reward is computed by evaluating the marginal contribution of individual item placements via counterfactual swaps (top right). A critic predicts both the expected interaction reward and its uncertainty for unobserved interactions. These estimates are used to form uncertainty-aware advantages, down-weighting unreliable signals during optimization (bottom).
  • Figure 2: Performance across all three needs on KuaiRec. FlexRec is trained jointly on all needs. Joint training yields consistently stronger performance across needs, supporting FlexRec as an all-purpose recommender conditioned by need instructions.
  • Figure 3: Validation NDCG@5 and NDCG@10 during training under different reward designs, showing that FlexRec's swap-based item-level reward outperforms sequence-level GRPO.
  • Figure 4: Validation NDCG@5 and NDCG@10 during training under different reward signals (UA denotes uncertainty-aware update), showing that a learned critic with uncertainty-aware GRPO achieves the strongest and most stable learning.