Taskmaster-1: Toward a Realistic and Diverse Dialog Dataset

TL;DR

Taskmaster-1 tackles the scarcity of high-quality goal-oriented dialog data by releasing a 13,215-dialog corpus across six domains, collected via two methods: two-person WOz-spoken dialogs and self-dialogs. It introduces a simple API-argument labeling scheme and demonstrates that Taskmaster-1 offers richer language and real-world entities than prior datasets like MultiWOZ. Baseline experiments with seq2seq architectures and Transformer-based models show strong automatic and human-aligned performance, and API-argument prediction benefits from a copying mechanism. The dual collection approaches provide complementary strengths in realism and scalability, making Taskmaster-1 a valuable resource for training and evaluating dialog systems and informing design choices around spoken vs written language and error handling.

Abstract

A significant barrier to progress in data-driven approaches to building dialog systems is the lack of high quality, goal-oriented conversational data. To help satisfy this elementary requirement, we introduce the initial release of the Taskmaster-1 dataset which includes 13,215 task-based dialogs comprising six domains. Two procedures were used to create this collection, each with unique advantages. The first involves a two-person, spoken "Wizard of Oz" (WOz) approach in which trained agents and crowdsourced workers interact to complete the task while the second is "self-dialog" in which crowdsourced workers write the entire dialog themselves. We do not restrict the workers to detailed scripts or to a small knowledge base and hence we observe that our dataset contains more realistic and diverse conversations in comparison to existing datasets. We offer several baseline models including state of the art neural seq2seq architectures with benchmark performance as well as qualitative human evaluations. Dialogs are labeled with API calls and arguments, a simple and cost effective approach which avoids the requirement of complex annotation schema. The layer of abstraction between the dialog model and the service provider API allows for a given model to interact with multiple services that provide similar functionally. Finally, the dataset will evoke interest in written vs. spoken language, discourse patterns, error handling and other linguistic phenomena related to dialog system research, development and design.

Figures (6)

• Figure 1: Sample Taskmaster-1 two-person dialog
• Figure 2: Sample instructions for agents playing "assistant" role
• Figure 3: Sample instructions for crowdsourced workers playing "user" role
• Figure 4: Sample instructions for written "self-dialogs"
• Figure 5: Sample one-person, written dialog