Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning

Xiao Yu, Maximillian Chen, Zhou Yu

TL;DR

<3-5 sentence high-level summary> GDP-Zero introduces a zero-training, open-loop MCTS framework for goal-oriented dialogue planning that uses prompting to an LLM to simulate policy priors, value functions, and user/system models at decision time. By treating dialogue planning as a stochastic MDP and regenerating utterances during traversal, it mitigates compounding errors common in dialogue simulations and achieves strong end-to-end performance on PersuasionForGood, outperforming prompting baselines and the RAP planner in both static and interactive evaluations. The approach demonstrates significant potential for data-scarce, high-stakes dialogue tasks, while openly acknowledging runtime and simulation-quality trade-offs and ethical considerations around persuasive AI agencies.

Abstract

Planning for goal-oriented dialogue often requires simulating future dialogue interactions and estimating task progress. Many approaches thus consider training neural networks to perform look-ahead search algorithms such as A* search and Monte Carlo Tree Search (MCTS). However, this training often requires abundant annotated data, which creates challenges when faced with noisy annotations or low-resource settings. We introduce GDP-Zero, an approach using Open-Loop MCTS to perform goal-oriented dialogue policy planning without any model training. GDP-Zero prompts a large language model to act as a policy prior, value function, user simulator, and system model during the tree search. We evaluate GDP-Zero on the goal-oriented task PersuasionForGood, and find that its responses are preferred over ChatGPT up to 59.32% of the time, and are rated more persuasive than ChatGPT during interactive evaluations.

Prompt-Based Monte-Carlo Tree Search for Goal-Oriented Dialogue Policy Planning

TL;DR

<3-5 sentence high-level summary> GDP-Zero introduces a zero-training, open-loop MCTS framework for goal-oriented dialogue planning that uses prompting to an LLM to simulate policy priors, value functions, and user/system models at decision time. By treating dialogue planning as a stochastic MDP and regenerating utterances during traversal, it mitigates compounding errors common in dialogue simulations and achieves strong end-to-end performance on PersuasionForGood, outperforming prompting baselines and the RAP planner in both static and interactive evaluations. The approach demonstrates significant potential for data-scarce, high-stakes dialogue tasks, while openly acknowledging runtime and simulation-quality trade-offs and ethical considerations around persuasive AI agencies.

Abstract

Planning for goal-oriented dialogue often requires simulating future dialogue interactions and estimating task progress. Many approaches thus consider training neural networks to perform look-ahead search algorithms such as A* search and Monte Carlo Tree Search (MCTS). However, this training often requires abundant annotated data, which creates challenges when faced with noisy annotations or low-resource settings. We introduce GDP-Zero, an approach using Open-Loop MCTS to perform goal-oriented dialogue policy planning without any model training. GDP-Zero prompts a large language model to act as a policy prior, value function, user simulator, and system model during the tree search. We evaluate GDP-Zero on the goal-oriented task PersuasionForGood, and find that its responses are preferred over ChatGPT up to 59.32% of the time, and are rated more persuasive than ChatGPT during interactive evaluations.
Paper Structure (44 sections, 5 equations, 3 figures, 19 tables, 1 algorithm)

This paper contains 44 sections, 5 equations, 3 figures, 19 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Prompting Methods
  4. Dialogue Policy Planning
  5. Method
  6. Problem Definition
  7. Dialogue Planning as a Stochastic MDP
  8. GDP-Zero
  9. Selection
  10. Expansion
  11. Evaluation
  12. Backpropagation
  13. Prediction
  14. Experiments
  15. Static Evaluation
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Using GDP-Zero for persuasion with zero model training.
  • Figure 2: GDP-Zero with ChatGPT backbone. During Selection, simulations are either sampled from cache or newly generated. During Expansion and Evaluation, we prompt ChatGPT for prior policy $\pi$ and value estimation.
  • Figure A1: Distribution of planned dialogue actions for each planner during interactive evaluations.