Plug-and-Play Policy Planner for Large Language Model Powered Dialogue Agents

TL;DR

<3-5 sentence high-level summary> The paper tackles the challenge of proactive dialogue by introducing a learnable plug-in policy planner (PPDPP) that augments LLM-powered agents. It combines supervised fine-tuning and reinforcement learning from goal-oriented AI feedback, using self-play with dual LLMs and an LLM reward model to optimize long-horizon dialogue goals. The plug-in is designed to be transferable across applications by swapping the plug-in without modifying the backbone LLM, and an interactive evaluation framework assesses policy planning effectiveness. Across negotiation, emotional support, and tutoring tasks, PPDPP outperforms prompting-based and prior RL-from-feedback baselines, demonstrating improved goal completion efficiency and effectiveness.

Abstract

Proactive dialogues serve as a practical yet challenging dialogue problem in the era of large language models (LLMs), where the dialogue policy planning is the key to improving the proactivity of LLMs. Most existing studies enable the dialogue policy planning of LLMs using various prompting schemes or iteratively enhance this capability in handling the given case with verbal AI feedback. However, these approaches are either bounded by the policy planning capability of the frozen LLMs or hard to be transferred to new cases. In this work, we introduce a new dialogue policy planning paradigm to strategize LLMs for proactive dialogue problems with a tunable language model plug-in as a plug-and-play dialogue policy planner, named PPDPP. Specifically, we develop a novel training framework to facilitate supervised fine-tuning over available human-annotated data as well as reinforcement learning from goal-oriented AI feedback with dynamic interaction data collected by the LLM-based self-play simulation. In this manner, the LLM-powered dialogue agent can not only be generalized to different cases after the training, but also be applicable to different applications by just substituting the learned plug-in. In addition, we propose to evaluate the policy planning capability of dialogue systems under the interactive setting. Experimental results demonstrate that PPDPP consistently and substantially outperforms existing approaches on three different proactive dialogue applications, including negotiation, emotional support, and tutoring dialogues.

Figures (5)

• Figure 1: The architectures of two types of LLM-based proactive dialogue systems. Dashed lines will be blocked during the inference phase.
• Figure 2: Comparisons of relative success rate against Standard at different conversation turns. The relative success rate is calculated by subtracting the actual success rate of the Standard prompting method from that of the concerned method.
• Figure 3: Comparisons of relative Sale-to-List Ratio against Standard at different turns (same legends as Figure \ref{['fig:exp-turn']}).
• Figure 4: Testing performance curve along with training episodes w.r.t different LLMs.
• Figure 5: Analysis of LLMs as reward model.