A LLM-based Controllable, Scalable, Human-Involved User Simulator Framework for Conversational Recommender Systems

TL;DR

This work tackles the challenge of evaluating Conversational Recommender Systems (CRS) with realistic and trustworthy user simulators. It introduces the Controllable, Scalable, and Human-Involved (CSHI) framework, a plugin-based, memory-augmented LLM-driven simulator that separates user modeling into three stages (User Profile Init, Preferences Init, and Message Handling) and imposes memory, anonymization, and known/unknown preference segmentation to curb data leakage. The approach demonstrates improved realism and controllability over single-prompt baselines across two CRS scenarios and datasets, including ReDial, OpenDialKG, and MovieLens, with qualitative case studies highlighting human-in-the-loop benefits. These results indicate CSHI enables more reliable CRS evaluation and the generation of higher-quality conversational recommendation datasets, with potential extensions to multimodal inputs.

Abstract

Conversational Recommender System (CRS) leverages real-time feedback from users to dynamically model their preferences, thereby enhancing the system's ability to provide personalized recommendations and improving the overall user experience. CRS has demonstrated significant promise, prompting researchers to concentrate their efforts on developing user simulators that are both more realistic and trustworthy. The emergence of Large Language Models (LLMs) has marked the onset of a new epoch in computational capabilities, exhibiting human-level intelligence in various tasks. Research efforts have been made to utilize LLMs for building user simulators to evaluate the performance of CRS. Although these efforts showcase innovation, they are accompanied by certain limitations. In this work, we introduce a Controllable, Scalable, and Human-Involved (CSHI) simulator framework that manages the behavior of user simulators across various stages via a plugin manager. CSHI customizes the simulation of user behavior and interactions to provide a more lifelike and convincing user interaction experience. Through experiments and case studies in two conversational recommendation scenarios, we show that our framework can adapt to a variety of conversational recommendation settings and effectively simulate users' personalized preferences. Consequently, our simulator is able to generate feedback that closely mirrors that of real users. This facilitates a reliable assessment of existing CRS studies and promotes the creation of high-quality conversational recommendation datasets.

Figures (10)

• Figure 1: CSHI framework overview.
• Figure 2: Workflow of the User Simulator.
• Figure 3: Percentage of successful recommendations by turn when constructing user simulators using CSHI.
• Figure 4: Recommendation accuracy across different rounds.
• Figure 5: Feedback from Different Simulators.