EasyRL4Rec: An Easy-to-use Library for Reinforcement Learning Based Recommender Systems

TL;DR

This work tackles the usability, evaluation standardization, and reproducibility challenges of reinforcement-learning-based recommender systems by introducing EasyRL4Rec, a lightweight library built around four core modules (Environment, Policy, StateTracker, Collector) and a unified training/evaluation pipeline. It supports both offline-log and user-model training, handles discrete and continuous actions via a conversion layer, and uses multiple StateTrackers to model user states, with offline MF-based evaluation to simulate feedback. The library is validated through extensive experiments across Coat, MovieLens, and KuaiRec, comparing model-free and batch RL methods, analyzing state-tracker choices, and assessing buffer-construction effects, thereby providing reproducible baselines and deeper insights into RL-based RS behavior. Overall, EasyRL4Rec lowers the barrier to developing and evaluating RL-based RSs, facilitating reproducibility, comparability, and rapid prototyping for future research and applications.

Abstract

Reinforcement Learning (RL)-Based Recommender Systems (RSs) have gained rising attention for their potential to enhance long-term user engagement. However, research in this field faces challenges, including the lack of user-friendly frameworks, inconsistent evaluation metrics, and difficulties in reproducing existing studies. To tackle these issues, we introduce EasyRL4Rec, an easy-to-use code library designed specifically for RL-based RSs. This library provides lightweight and diverse RL environments based on five public datasets and includes core modules with rich options, simplifying model development. It provides unified evaluation standards focusing on long-term outcomes and offers tailored designs for state modeling and action representation for recommendation scenarios. Furthermore, we share our findings from insightful experiments with current methods. EasyRL4Rec seeks to facilitate the model development and experimental process in the domain of RL-based RSs. The library is available for public use.

Figures (5)

• Figure 1: Architecture of EasyRL4Rec. The library is structured around four core modules: Environment (abbreviated as Envs), Policy, StateTracker, and Collector. Buffer serves as a fundamental data structure for organizing raw data trajectories, while the Trainer and Evaluator act as executors, managing the entire process.
• Figure 2: Visualization of data/trajectories stored in Buffer. To support simultaneous interactions in multiple environments, Buffer comprises interaction data from $n$ environments, with different trajectories in each environment represented by distinct colors and the presence of the start symbol.
• Figure 3: Two Training Settings. Blue lines represent the process of learning from offline logs, while red lines represent the process of learning with a user model.
• Figure 4: Variation curves of 1) cumulative reward, 2) interaction length, and 3) single-round reward during training on the Coat and KuaiRec datasets.
• Figure 5: Demonstration of preference overestimation issue on MovieLens-1M, with red lines representing -MSE of User Model, while orange bars and blue bars representing estimated reward and true reward respectively.