Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems

TL;DR

This work proposes SUIT (SUbgoal-aware ITerative Training), an iterative training approach for improving ToD systems that reaches new state-of-the-art performance on a popular ToD benchmark.

Abstract

Task-oriented Dialog (ToD) systems have to solve multiple subgoals to accomplish user goals, whereas feedback is often obtained only at the end of the dialog. In this work, we propose SUIT (SUbgoal-aware ITerative Training), an iterative training approach for improving ToD systems. We sample dialogs from the model we aim to improve and determine subgoals that contribute to dialog success using distant supervision to obtain high quality training samples. We show how this data improves supervised fine-tuning or, alternatively, preference learning results. SUIT is able to iteratively generate more data instead of relying on fixed static sets. SUIT reaches new state-of-the-art performance on a popular ToD benchmark.

Figures (3)

• Figure 1: Successful dialog example.
• Figure 2: Overview of training procedure in Suit. We sample multiple dialogs for one user goal, where each dialog $D_s$ consists of user turns $U_{st}$, and system turns, which are split into dialog states $B_{st}$, system actions $A_{st}$ and responses $R_{st}$. We evaluate dialog success at the end of each generated dialog. For every successful dialog $D_s$, we replace parts of system turns (subgoals) with the respective parts coming from wrong dialogs $D_{o,j,u}$. If the dialog success flips to unsuccessful, we add the successful subgoal as training data.
• Figure 3: Example for input/ouput representation in Suit.