Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Chitchat as Interference: Adding User Backstories to Task-Oriented Dialogues

Armand Stricker, Patrick Paroubek

TL;DR

Chitchat in task-oriented dialogues is treated as a natural interference that can derail task progress. The authors automate the generation of user backstories that accompany task requests using few-shot prompting with the open-source Llama-2-70B, and augment MultiWOZ to create more challenging TOD scenarios. They propose a four-step augmentation pipeline and evaluate three baselines, finding that a model trained on augmented data can acknowledge backstories while maintaining task momentum, with human judges favoring the augmented-trained system in many cases. This work provides a practical, scalable method for creating diverse chitchat TOD examples to test and strengthen system resilience in real-world conversations.

Abstract

During task-oriented dialogues (TODs), human users naturally introduce chitchat that is beyond the immediate scope of the task, interfering with the flow of the conversation. To address this issue without the need for expensive manual data creation, we use few-shot prompting with Llama-2-70B to enhance the MultiWOZ dataset with user backstories, a typical example of chitchat interference in TODs. We assess the impact of this addition by testing two models: one trained solely on TODs and another trained on TODs with a preliminary chitchat interaction. Our analysis demonstrates that our enhanced dataset poses a challenge for these systems. Moreover, we demonstrate that our dataset can be effectively used for training purposes, enabling a system to consistently acknowledge the user's backstory while also successfully moving the task forward in the same turn, as confirmed by human evaluation. These findings highlight the benefits of generating novel chitchat-TOD scenarios to test TOD systems more thoroughly and improve their resilience to natural user interferences

Chitchat as Interference: Adding User Backstories to Task-Oriented Dialogues

TL;DR

Chitchat in task-oriented dialogues is treated as a natural interference that can derail task progress. The authors automate the generation of user backstories that accompany task requests using few-shot prompting with the open-source Llama-2-70B, and augment MultiWOZ to create more challenging TOD scenarios. They propose a four-step augmentation pipeline and evaluate three baselines, finding that a model trained on augmented data can acknowledge backstories while maintaining task momentum, with human judges favoring the augmented-trained system in many cases. This work provides a practical, scalable method for creating diverse chitchat TOD examples to test and strengthen system resilience in real-world conversations.

Abstract

During task-oriented dialogues (TODs), human users naturally introduce chitchat that is beyond the immediate scope of the task, interfering with the flow of the conversation. To address this issue without the need for expensive manual data creation, we use few-shot prompting with Llama-2-70B to enhance the MultiWOZ dataset with user backstories, a typical example of chitchat interference in TODs. We assess the impact of this addition by testing two models: one trained solely on TODs and another trained on TODs with a preliminary chitchat interaction. Our analysis demonstrates that our enhanced dataset poses a challenge for these systems. Moreover, we demonstrate that our dataset can be effectively used for training purposes, enabling a system to consistently acknowledge the user's backstory while also successfully moving the task forward in the same turn, as confirmed by human evaluation. These findings highlight the benefits of generating novel chitchat-TOD scenarios to test TOD systems more thoroughly and improve their resilience to natural user interferences
Paper Structure (22 sections, 4 figures, 9 tables)

This paper contains 22 sections, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Proposed Augmentation Pipeline
  3. Seed Situation
  4. Turn Selection
  5. Exchange Augmentation
  6. Automatic filtering
  7. Human Evaluation
  8. Methodology and Experimental Setup
  9. Methodology
  10. Experimental Setup
  11. Implementation
  12. Automatic Evaluation Metrics
  13. Results and Discussion
  14. Automatic Evaluation Results
  15. Human Evaluation Results
  16. ...and 7 more sections

Figures (4)

  • Figure 1: A chatty user incorporates elements of backstory to their task-oriented request, creating a natural interference in the TOD interaction. The system reaction accommodates the additional backstory with support and understanding, all the while avoiding the introduction of new topics. This design choice ensures that the system seamlessly transitions back to the task at hand, effectively assisting the user in achieving their goal.
  • Figure 2: Our proposed augmentation generation pipeline. Given a Fusedchat dialogue, we 1) summarize the prepended chitchat into a seed situation, 2) select a random exchange to augment from the task continuation, 3) expand the exchange with in-context learning by first generating the backstory and then the chitchat reaction, 4) filter out potentially low quality generations.
  • Figure 3: Few-shot prompt (3 examples) for generating the seed situation. Prompts for generating the backstory and the reaction follow a similar structure.
  • Figure 4: An overview of the SimpleToD approach. At training time, a sequence of components is fed into a generative language model. During inference, only the dialogue history is used as input, and each component is generated in step-by-step autoregressive fashion.