Reformulating Conversational Recommender Systems as Tri-Phase Offline Policy Learning

TL;DR

The paper tackles the problem of overfitting and unrealistic evaluation in conversational recommender systems by introducing Tri-Phase Offline Policy Learning (TCRS). It fuses a Conversational User Model trained offline, PPO-based policy learning, and a controllable user simulation for robust offline training and realistic evaluation, decoupling training from evaluation. Key contributions include a multi-task CUM that predicts item and attribute preferences, a PPO policy framework with pruned action spaces, and a controllable, adaptive user simulator with diverse sampling strategies. The results across three datasets show that TCRS enhances robustness, adaptability, and recommendation accuracy, offering a more faithful evaluation environment and better alignment with evolving user preferences in real-world deployment.

Abstract

Existing Conversational Recommender Systems (CRS) predominantly utilize user simulators for training and evaluating recommendation policies. These simulators often oversimplify the complexity of user interactions by focusing solely on static item attributes, neglecting the rich, evolving preferences that characterize real-world user behavior. This limitation frequently leads to models that perform well in simulated environments but falter in actual deployment. Addressing these challenges, this paper introduces the Tri-Phase Offline Policy Learning-based Conversational Recommender System (TCRS), which significantly reduces dependency on real-time interactions and mitigates overfitting issues prevalent in traditional approaches. TCRS integrates a model-based offline learning strategy with a controllable user simulation that dynamically aligns with both personalized and evolving user preferences. Through comprehensive experiments, TCRS demonstrates enhanced robustness, adaptability, and accuracy in recommendations, outperforming traditional CRS models in diverse user scenarios. This approach not only provides a more realistic evaluation environment but also facilitates a deeper understanding of user behavior dynamics, thereby refining the recommendation process.

Figures (5)

• Figure 1: Comparative Frameworks in CRS Training and Evaluation
• Figure 2: User-centric vs. Item-centric Preference Modeling in Multi-round Conversational Recommendation Scenario
• Figure 3: The Tri-Phase Offline Policy Learning-based Conversational Recommender System (TCRS) Framework. Conversational User Model (CUM) is first trained on offline data to capture dynamic and personalized user preferences. The learned user model then serves as the simulated environment for Policy Learning, where the recommendation policy is optimized to maximize long-term user satisfaction. Finally, the trained policy is evaluated using an independent, controllable user simulator that can adapt to diverse user preference scenarios, enabling a comprehensive assessment of the policy's adaptability.
• Figure 4: Evaluating TCRS policies under varying personalized preference ($\alpha$) and preference evolution rate ($\Delta\lambda$) in the Controllable User Simulation (RQ3).
• Figure 5: Examining the alignment between user preference and target item attributes (match rate) during the conversational interaction, under varying levels of personalization ($\alpha$) in the LastFM and Yelp datasets (RQ4).