Reinforced Strategy Optimization for Conversational Recommender Systems via Network-of-Experts

TL;DR

This work tackles the challenge of explicit interaction-strategy optimization in LLM-based CRSs by introducing Reinforced Strategy Optimization (RSO), a hierarchical Network-of-Experts that separates macro-level planning from micro-level adaptation. A Planner selects high-level strategies, while an Actor, aided by a Preference Reasoner and a Fact Retriever, generates grounded, user-specific responses. The macro-level policy is learned through reinforcement learning guided by an LLM-based reward model, following a two-stage process of warm-up fine-tuning and entropy-regularized RL tuning. Experimental results on Inspired and ReDial show that RSO achieves state-of-the-art performance across conversational and recommendation metrics, and the 7B NoE variant matches or surpasses much larger baselines, demonstrating efficient, scalable strategy optimization for CRSs.

Abstract

Conversational Recommender Systems (CRSs) aim to provide personalized recommendations through multi-turn natural language interactions with users. Given the strong interaction and reasoning skills of Large Language Models (LLMs), leveraging LLMs for CRSs has recently emerged as a promising direction. However, existing LLM-based methods often lack explicit optimization of interaction strategies, instead relying on unified prompts and the LLM's internal knowledge to decide how to interact, which can lead to suboptimal outcomes. In this paper, we propose a novel Reinforced Strategy Optimization (RSO) method for CRS, which decomposes the process of generating strategy-driven response decisions into the macro-level strategy planning and micro-level strategy adaptation through a network-of-experts architecture. At the macro level, a Planner expert selects macro-level interaction strategies (e.g., recommend, explain, encourage). At the micro level, an Actor expert generates detailed responses conditioned on the selected macro-level strategy, guided by auxiliary experts that provide complementary information such as user preferences and factual grounding. This hierarchical decomposition disentangles the optimization of different sub-tasks involved in CRS response generation, enabling more tractable learning at each level. To address the scarcity of high-quality multi-turn training data, we formulate strategy learning as a reinforcement learning problem, guided by an LLM-based reward model to achieve automatic strategy exploration. Extensive experiments show that RSO significantly improves interaction performance compared to state-of-the-art baselines, demonstrating the effectiveness of explicit hierarchical strategy optimization for CRS.

Figures (6)

• Figure 1: Comparison between previous CRS strategies and our proposed strategy optimization. The right panel illustrates the difference in a single system response: previous methods rely on prompt-based predefined strategies pccrsinspired and yield generic responses, while our approach performs explicit strategy optimization to generate user-specific responses.
• Figure 2: Overview of our RSO framework. Specifically, the Planner performs macro-level strategy planning, while the Preference Reasoner, Fact Retriever, and Actor collaboratively realize micro-level strategy adaptation. The Rewarder provides a turn-level signal to update the Planner.
• Figure 3: Comparison of macro-level strategy distributions across conversation progress before (a) and after (b) Entropy-regularized RL tuning on the Inspired dataset. Entropy-regularized RL reduces over-reliance on a few dominant strategies and encourages broader, context-aware strategy exploration, resulting in more adaptive and balanced conversational behaviors.
• Figure 4: Comparison of our RSO (7B) with QwenCRS-14B and QwenCRS-32B on Inspired and ReDial datasets.
• Figure 5: Ablation studies on the impact of the Fact Retriever, Planner, and Preference Reasoner on INSPIRED and ReDial datasets. The Fact Retriever significantly improves Credibility, while both the Planner and Preference Reasoner are essential for achieving high Conversation Success Rate.