Search-Based Interaction For Conversation Recommendation via Generative Reward Model Based Simulated User

TL;DR

The paper tackles the challenge of extracting complex, multifaceted user preferences from short conversations in conversational recommender systems (CRSs). It introduces GRSU, a generative reward model-based simulated user with two feedback actions—generative item scoring and attribute-based item critique—trained via instruction tuning on synthetic data and integrated with a beam-search interaction framework. Interaction between CRSs and GRSU is formulated as a Markov decision process and guided by a reward derived from the simulated user's feedback, with an efficient ranking step to finalize recommendations. Experiments on ReDial and INSPIRED show that GRSU improves item ranking and overall CRS performance, demonstrates transferability to low-resource settings, and generalizes across various LLM-based CRSs. This approach enables scalable, label-free automatic interaction that reduces the need for real-user involvement while maintaining or improving recommendation quality.

Abstract

Conversational recommendation systems (CRSs) use multi-turn interaction to capture user preferences and provide personalized recommendations. A fundamental challenge in CRSs lies in effectively understanding user preferences from conversations. User preferences can be multifaceted and complex, posing significant challenges for accurate recommendations even with access to abundant external knowledge. While interaction with users can clarify their true preferences, frequent user involvement can lead to a degraded user experience. To address this problem, we propose a generative reward model based simulated user, named GRSU, for automatic interaction with CRSs. The simulated user provides feedback to the items recommended by CRSs, enabling them to better capture intricate user preferences through multi-turn interaction. Inspired by generative reward models, we design two types of feedback actions for the simulated user: i.e., generative item scoring, which offers coarse-grained feedback, and attribute-based item critique, which provides fine-grained feedback. To ensure seamless integration, these feedback actions are unified into an instruction-based format, allowing the development of a unified simulated user via instruction tuning on synthesized data. With this simulated user, automatic multi-turn interaction with CRSs can be effectively conducted. Furthermore, to strike a balance between effectiveness and efficiency, we draw inspiration from the paradigm of reward-guided search in complex reasoning tasks and employ beam search for the interaction process. On top of this, we propose an efficient candidate ranking method to improve the recommendation results derived from interaction. Extensive experiments on public datasets demonstrate the effectiveness, efficiency, and transferability of our approach.

Figures (3)

• Figure 1: Approach overview. (a) We use search-based interaction between CRSs and our simulated user to generate candidate items, which is followed by an efficient ranking method to derive the final item list. (b) A simplified state transition example: Starting from a current state (i.e., an item list), the simulated user provides attribute-based item critiques, which the CRS uses to refine its previous recommendations, thereby transitioning to the next state. (c) A simplified reward example: Given a current state (i.e., an item list), the simulated user provides generative item scoring. The probability of the reward token (i.e., "No") is extracted as the score for each item, and their average is the reward of the current state (i.e., the item list).
• Figure 2: Ablation study on the INSPIRED dataset. "critiquing" refers to attribute-based item critiquing, "scoring" refers to generative item scoring, and "ranking" refers to efficient candidate ranking.
• Figure 3: The performance changes with respect to the depth of search. As a reference, we add the performance of ReFICR.