Energy-Guided Diffusion Sampling for Long-Term User Behavior Prediction in Reinforcement Learning-based Recommendation

TL;DR

The paper tackles offline RL challenges in reinforcement learning-based recommender systems by introducing DAC4Rec, a diffusion-enhanced actor-critic framework. It uses a conditional reverse diffusion process to represent the behavior policy, augmented with a Q-value guided improvement term and an energy-based sampling mechanism to reduce randomness during evaluation. Key contributions include (i) a diffusion-based, expressive policy for long-term user preference modeling, (ii) a combined behavior-cloning and policy-improvement objective, and (iii) energy-guided sampling that integrates with existing RL backbones. Extensive experiments on six real-world offline datasets and online simulation demonstrate superior performance and robustness, with demonstrated generalization to SAC and DDPG backbones, indicating wide applicability to RL-based recommender systems.

Abstract

Reinforcement learning-based recommender systems (RL4RS) have gained attention for their ability to adapt to dynamic user preferences. However, these systems face challenges, particularly in offline settings, where data inefficiency and reliance on pre-collected trajectories limit their broader applicability. While offline reinforcement learning methods leverage extensive datasets to address these issues, they often struggle with noisy data and fail to capture long-term user preferences, resulting in suboptimal recommendation policies. To overcome these limitations, we propose Diffusion-enhanced Actor-Critic for Offline RL4RS (DAC4Rec), a novel framework that integrates diffusion processes with reinforcement learning to model complex user preferences more effectively. DAC4Rec leverages the denoising capabilities of diffusion models to enhance the robustness of offline RL algorithms and incorporates a Q-value-guided policy optimization strategy to better handle suboptimal trajectories. Additionally, we introduce an energy-based sampling strategy to reduce randomness during recommendation generation, ensuring more targeted and reliable outcomes. We validate the effectiveness of DAC4Rec through extensive experiments on six real-world offline datasets and in an online simulation environment, demonstrating its ability to optimize long-term user preferences. Furthermore, we show that the proposed diffusion policy can be seamlessly integrated into other commonly used RL algorithms in RL4RS, highlighting its versatility and wide applicability.

Figures (3)

• Figure 1: The illustration of the expressiveness challenges that were faced by offline RL4RS
• Figure 2: An overview of the offline RL4RS and the framework of the proposed DAC4Rec. The policy $\mu$ represents the proposed diffusion policy. While DAC4Rec is built on the actor-critic structure, it can be easily adapted to any Q-value-involved RL methods, including Q-learning, SAC, and DDPG (or TD3).
• Figure 3: (a). Overall comparison result with variance between the baselines and DAC4Rec in the VirtualTaobao simulation environment. (b).Hyperparameter diffusion step $N$ study, the value reported in the average CTR over $100,000$ timesteps. (c).Ablation study, the value reported in the average CTR over $100,000$ timesteps