Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue

Huifang Du, Shuqin Li, Minghao Wu, Xuejing Feng, Yuan-Fang Li, Haofen Wang

TL;DR

This work extends RL into both understanding and generation tasks by introducing step-by-step rewards throughout the token generation, and shows superior few-shot ability in low-resource settings compared to current models.

Abstract

Reinforcement learning (RL) is a powerful approach to enhance task-oriented dialogue (TOD) systems. However, existing RL methods tend to mainly focus on generation tasks, such as dialogue policy learning (DPL) or response generation (RG), while neglecting dialogue state tracking (DST) for understanding. This narrow focus limits the systems to achieve globally optimal performance by overlooking the interdependence between understanding and generation. Additionally, RL methods face challenges with sparse and delayed rewards, which complicates training and optimization. To address these issues, we extend RL into both understanding and generation tasks by introducing step-by-step rewards throughout the token generation. The understanding reward increases as more slots are correctly filled in DST, while the generation reward grows with the accurate inclusion of user requests. Our approach provides a balanced optimization aligned with task completion. Experimental results demonstrate that our approach effectively enhances the performance of TOD systems and achieves new state-of-the-art results on three widely used datasets, including MultiWOZ2.0, MultiWOZ2.1, and In-Car. Our approach also shows superior few-shot ability in low-resource settings compared to current models.

Rewarding What Matters: Step-by-Step Reinforcement Learning for Task-Oriented Dialogue

TL;DR

This work extends RL into both understanding and generation tasks by introducing step-by-step rewards throughout the token generation, and shows superior few-shot ability in low-resource settings compared to current models.

Abstract

Reinforcement learning (RL) is a powerful approach to enhance task-oriented dialogue (TOD) systems. However, existing RL methods tend to mainly focus on generation tasks, such as dialogue policy learning (DPL) or response generation (RG), while neglecting dialogue state tracking (DST) for understanding. This narrow focus limits the systems to achieve globally optimal performance by overlooking the interdependence between understanding and generation. Additionally, RL methods face challenges with sparse and delayed rewards, which complicates training and optimization. To address these issues, we extend RL into both understanding and generation tasks by introducing step-by-step rewards throughout the token generation. The understanding reward increases as more slots are correctly filled in DST, while the generation reward grows with the accurate inclusion of user requests. Our approach provides a balanced optimization aligned with task completion. Experimental results demonstrate that our approach effectively enhances the performance of TOD systems and achieves new state-of-the-art results on three widely used datasets, including MultiWOZ2.0, MultiWOZ2.1, and In-Car. Our approach also shows superior few-shot ability in low-resource settings compared to current models.
Paper Structure (36 sections, 7 equations, 8 figures, 3 tables)

This paper contains 36 sections, 7 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Pipeline and End-to-End Approaches.
  4. RL-Based Policy Learning.
  5. Reward Design for TOD.
  6. Large Language Models for TOD.
  7. Preliminary
  8. Supervised Fine-Tuning for TOD
  9. Reinforcement Learning for TOD
  10. Main Method
  11. Task Completion Metrics
  12. Step-by-Step Goal-Oriented Reward
  13. Understanding Reward.
  14. Generation Reward.
  15. TOD Reward.
  16. ...and 21 more sections

Figures (8)

  • Figure 1: A task-oriented dialogue system needs to successfully perform both understanding and generation to achieve its dialogue goals.
  • Figure 2: Overview of our approach. Left: We use the masked policy to optimize understanding and generation end-to-end with our reward function for TOD systems. Context is the concatenation of belief state (BS), dialogue act (DA), and system response (SR) at the previous turn. Special characters like <sos_$b$>, <sos_$a$>, and <sos_$r$> denote the start of BS, DA, and SR, while <eos_$b$>, <eos_$a$>, and <eos_$r$> denote their endings. Right: The designed reward function provides step-by-step rewards for understanding and generation tasks. $Reward_{u}$ refers to \ref{['eq:understanding']}, $Reward_{g}$ refers to \ref{['eq:generation']}, and $Reward_{tod}$ refers to \ref{['eq:tod']}.
  • Figure 3: Results of low-resource experiments. 5% (121 dialogues), 10% (242 dialogues), 20% (485 dialogues), 30% (727 dialogues), 40% (970 dialogues), and 50% (1212 dialogues) of training data is used to train each model. Results are shown as mean values over five runs.
  • Figure 4: The human evaluation results regarding appropriateness and fluency. The numbers represent the average and the standard deviation for each method.
  • Figure 5: Reward accumulation for different tasks: DST+DPL, DST+RG, and DST+DPL+RG during token generation. Our reward function progressively provides important feedback for understanding and generation tasks.
  • ...and 3 more figures