Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A General Highly Accurate Online Planning Method Integrating Large Language Models into Nested Rollout Policy Adaptation for Dialogue Tasks

Hui Wang, Fafa Zhang, Xiaoyu Zhang, Chaoxu Mu

TL;DR

<3-5 sentence high-level summary> NRPA-GD presents an online planning framework for goal-oriented dialogue that eliminates offline policy training by integrating a Nested Rollout Policy Adaptation (NRPA) mechanism with a Large Language Model to simulate user and system behavior. The method treats dialogue as an MDP and uses a two-level nested NMCS-inspired planner where policy weights are iteratively updated to favor high-reward action sequences, enabling efficient, zero-shot planning. Across four datasets encompassing emotional support, tutoring, bargaining, and persuasion, NRPA-GD outperforms prompt-engineering and offline RL baselines and even matches or surpasses larger pre-trained models, with strong results on moderate-size LLMs. The work demonstrates the viability and advantages of planning-on-LLMs for practical dialogue tasks, offering scalable, training-free policy optimization and highlighting avenues for further efficiency improvements.

Abstract

In goal-oriented dialogue tasks, the main challenge is to steer the interaction towards a given goal within a limited number of turns. Existing approaches either rely on elaborate prompt engineering, whose effectiveness is heavily dependent on human experience, or integrate policy networks and pre-trained policy models, which are usually difficult to adapt to new dialogue scenarios and costly to train. Therefore, in this paper, we present Nested Rollout Policy Adaptation for Goal-oriented Dialogue (NRPA-GD), a novel dialogue policy planning method that completely avoids specific model training by utilizing a Large Language Model (LLM) to simulate behaviors of user and system at the same time. Specifically, NRPA-GD constructs a complete evaluation mechanism for dialogue trajectories and employs an optimization framework of nested Monte Carlo simulation and policy self-adaptation to dynamically adjust policies during the dialogue process. The experimental results on four typical goal-oriented dialogue datasets show that NRPA-GD outperforms both existing prompt engineering and specifically pre-trained model-based methods. Impressively, NRPA-GD surpasses ChatGPT and pre-trained policy models with only a 0.6-billion-parameter LLM. The proposed approach further demonstrates the advantages and novelty of employing planning methods on LLMs to solve practical planning tasks.

A General Highly Accurate Online Planning Method Integrating Large Language Models into Nested Rollout Policy Adaptation for Dialogue Tasks

TL;DR

<3-5 sentence high-level summary> NRPA-GD presents an online planning framework for goal-oriented dialogue that eliminates offline policy training by integrating a Nested Rollout Policy Adaptation (NRPA) mechanism with a Large Language Model to simulate user and system behavior. The method treats dialogue as an MDP and uses a two-level nested NMCS-inspired planner where policy weights are iteratively updated to favor high-reward action sequences, enabling efficient, zero-shot planning. Across four datasets encompassing emotional support, tutoring, bargaining, and persuasion, NRPA-GD outperforms prompt-engineering and offline RL baselines and even matches or surpasses larger pre-trained models, with strong results on moderate-size LLMs. The work demonstrates the viability and advantages of planning-on-LLMs for practical dialogue tasks, offering scalable, training-free policy optimization and highlighting avenues for further efficiency improvements.

Abstract

In goal-oriented dialogue tasks, the main challenge is to steer the interaction towards a given goal within a limited number of turns. Existing approaches either rely on elaborate prompt engineering, whose effectiveness is heavily dependent on human experience, or integrate policy networks and pre-trained policy models, which are usually difficult to adapt to new dialogue scenarios and costly to train. Therefore, in this paper, we present Nested Rollout Policy Adaptation for Goal-oriented Dialogue (NRPA-GD), a novel dialogue policy planning method that completely avoids specific model training by utilizing a Large Language Model (LLM) to simulate behaviors of user and system at the same time. Specifically, NRPA-GD constructs a complete evaluation mechanism for dialogue trajectories and employs an optimization framework of nested Monte Carlo simulation and policy self-adaptation to dynamically adjust policies during the dialogue process. The experimental results on four typical goal-oriented dialogue datasets show that NRPA-GD outperforms both existing prompt engineering and specifically pre-trained model-based methods. Impressively, NRPA-GD surpasses ChatGPT and pre-trained policy models with only a 0.6-billion-parameter LLM. The proposed approach further demonstrates the advantages and novelty of employing planning methods on LLMs to solve practical planning tasks.

Paper Structure

This paper contains 29 sections, 5 equations, 3 figures, 9 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Method
  4. Problem Definition
  5. NRPA Planner
  6. Experiments
  7. Datasets
  8. Baselines
  9. Evaluation Metrics
  10. Results and Analysis
  11. Static Evaluation
  12. Automatic Evaluation
  13. Performance on Different LLMs
  14. Human Evaluation
  15. Conclusion
  16. ...and 14 more sections

Figures (3)

  • Figure 1: The NRPA-GD system uses a nested architecture to implement recursive policy optimization through two levels. In level 2, simulations are performed by recursively calling level 1 to optimize the policy distribution based on the reward values generated by policy choices, and the updated policies are passed to the next round of simulation iterations.
  • Figure 2: Human evaluation results on P4G and CIMA.
  • Figure 3: Human evaluation results on ESConv and CraigslistBargain.