User Simulator-Guided Multi-Turn Preference Optimization for Reasoning LLM-based Conversational Recommendation

Abstract

Conversational Recommender Systems (CRSs) leverage natural language interactions for personalized recommendation, yet information-scarce dialogue histories and single-turn recommendation paradigms may severely hinder accurate modeling of complex user preferences. To alleviate this issue, recent studies have introduced LLM-based user simulators, which generate natural language feedback and perform simulated multi-turn interactions to assist recommendation. Nevertheless, since simulators cannot access true user preference labels during inference, their feedback may deviate from actual user interests, causing errors to accumulate over multiple interactions and severely affecting the generalization of the recommender. Inspired by the multi-step reasoning capabilities of LLMs and the effectiveness of reinforcement learning in policy optimization, we propose SMTPO, a user simulator-guided multi-turn preference optimization conversational recommendation framework. To align simulator-generated feedback with true user preferences in the absence of explicit labels, we enhance feedback quality via multi-task supervised fine-tuning (SFT), enabling the simulator to better reflect users' complex and diverse needs. To address the challenge of biased feedback destabilizing multi-turn optimization, we first allow the reasoning LLM-based recommender to learn preference reasoning and recommendation patterns through SFT and then employ reinforcement learning with fine-grained reward design to progressively align with true user preferences, improving recommendation performance. Extensive experiments on public datasets demonstrate the effectiveness and transferability of our method.

Figures (4)

• Figure 1: The previous method frozen recommender parameters, relied on beam search and simulator-filtered results, making it sensitive to feedback bias. Our method trains the SFT-initialized recommender via RL to achieve robust and accurate preference modeling and recommendation.
• Figure 2: Overview of SMTPO training and multi-turn interaction: (1) The multi-turn interaction process among the recommender, simulator, and retriever. (2) The recommender is trained first with SFT and then with multi-turn RL. (3) The simulator is trained via multi-task SFT. (4) The retriever is obtained using collaborative–semantic dual-view modeling.
• Figure 3: The performance impact of different rewards on the INSPIRED dataset hayati2020inspired.
• Figure 4: Impact of multi-turn preference optimization on recommendation performance.