DARLR: Dual-Agent Offline Reinforcement Learning for Recommender Systems with Dynamic Reward

TL;DR

DARLR tackles reward inaccuracies in model-based offline reinforcement learning for recommender systems by introducing a dual-agent framework: a selector that identifies representative reference users and a recommender that uses those references to iteratively refine rewards and adapt uncertainty penalties. The selector enables dynamic reward shaping and a corresponding dynamic uncertainty design, while the recommender aggregates reference information to improve both reward predictions and exploration-exploitation balance. Empirical results on four benchmarks show that DARLR consistently outperforms baselines and closely approaches the ideal GT reward scenario, validating the effectiveness of dynamic reward shaping in offline RecSys. The work offers a practical approach to enhance policy learning under imperfect world models and suggests avenues for scalable dynamic reward shaping in large-scale systems.

Abstract

Model-based offline reinforcement learning (RL) has emerged as a promising approach for recommender systems, enabling effective policy learning by interacting with frozen world models. However, the reward functions in these world models, trained on sparse offline logs, often suffer from inaccuracies. Specifically, existing methods face two major limitations in addressing this challenge: (1) deterministic use of reward functions as static look-up tables, which propagates inaccuracies during policy learning, and (2) static uncertainty designs that fail to effectively capture decision risks and mitigate the impact of these inaccuracies. In this work, a dual-agent framework, DARLR, is proposed to dynamically update world models to enhance recommendation policies. To achieve this, a \textbf{\textit{selector}} is introduced to identify reference users by balancing similarity and diversity so that the \textbf{\textit{recommender}} can aggregate information from these users and iteratively refine reward estimations for dynamic reward shaping. Further, the statistical features of the selected users guide the dynamic adaptation of an uncertainty penalty to better align with evolving recommendation requirements. Extensive experiments on four benchmark datasets demonstrate the superior performance of DARLR, validating its effectiveness. The code is available at https://github.com/ArronDZhang/DARLR.

Figures (5)

• Figure 1: Recommendation policies trained from scratch with dynamic reward functions converge better than those with static reward functions on KuaiRand gao2022kuairand and KuaiRec gao2022kuairec.
• Figure 2: Overall Framework of DARLR with three main modules: a world model, a selector agent and a recommender agent. The gradient flow will influence the training of both agents, as well as the update of the world model.
• Figure 3: The overall performance on KuaiRand and KuaiRec.
• Figure 4: The mean reward differences (Error) during training of DORL, ROLeR and DARLR. The reward differences of the dynamic reward shaping method (DARLR) are consistently smaller than those of the static reward shaping method (ROLeR and DORL).
• Figure 5: Hyperparameter sensitivity with different $K^{\text{sel}}$, $\lambda_s$, $\lambda_d$, $\lambda_{U}$, and $\lambda_{E}$ on four datasets.