Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

End-to-end Conversation Modeling Track in DSTC6

Chiori Hori, Takaaki Hori

TL;DR

This DSTC6 paper investigates end-to-end conversation modeling using human-to-human data to emulate customer-service dialog agents. It defines a Twitter-based mandatory task and an OpenSubtitles-based optional task, outlines data collection and preprocessing, surveys a range of submitted systems, and evaluates them with both automatic metrics and human judgments. The results reveal discrepancies between objective scores and perceived quality, suggesting the need for more robust evaluation methods and objective functions. Overall, the work demonstrates progress in end-to-end, knowledge-informed dialogue and highlights directions for improving evaluation in this domain.

Abstract

End-to-end training of neural networks is a promising approach to automatic construction of dialog systems using a human-to-human dialog corpus. Recently, Vinyals et al. tested neural conversation models using OpenSubtitles. Lowe et al. released the Ubuntu Dialogue Corpus for researching unstructured multi-turn dialogue systems. Furthermore, the approach has been extended to accomplish task oriented dialogs to provide information properly with natural conversation. For example, Ghazvininejad et al. proposed a knowledge grounded neural conversation model [3], where the research is aiming at combining conversational dialogs with task-oriented knowledge using unstructured data such as Twitter data for conversation and Foursquare data for external knowledge.However, the task is still limited to a restaurant information service, and has not yet been tested with a wide variety of dialog tasks. In addition, it is still unclear how to create intelligent dialog systems that can respond like a human agent. In consideration of these problems, we proposed a challenge track to the 6th dialog system technology challenges (DSTC6) using human-to-human dialog data to mimic human dialog behaviors. The focus of the challenge track is to train end-to-end conversation models from human-to-human conversation and accomplish end-to-end dialog tasks in various situations assuming a customer service, in which a system plays a role of human agent and generates natural and informative sentences in response to user's questions or comments given dialog context.

End-to-end Conversation Modeling Track in DSTC6

TL;DR

This DSTC6 paper investigates end-to-end conversation modeling using human-to-human data to emulate customer-service dialog agents. It defines a Twitter-based mandatory task and an OpenSubtitles-based optional task, outlines data collection and preprocessing, surveys a range of submitted systems, and evaluates them with both automatic metrics and human judgments. The results reveal discrepancies between objective scores and perceived quality, suggesting the need for more robust evaluation methods and objective functions. Overall, the work demonstrates progress in end-to-end, knowledge-informed dialogue and highlights directions for improving evaluation in this domain.

Abstract

End-to-end training of neural networks is a promising approach to automatic construction of dialog systems using a human-to-human dialog corpus. Recently, Vinyals et al. tested neural conversation models using OpenSubtitles. Lowe et al. released the Ubuntu Dialogue Corpus for researching unstructured multi-turn dialogue systems. Furthermore, the approach has been extended to accomplish task oriented dialogs to provide information properly with natural conversation. For example, Ghazvininejad et al. proposed a knowledge grounded neural conversation model [3], where the research is aiming at combining conversational dialogs with task-oriented knowledge using unstructured data such as Twitter data for conversation and Foursquare data for external knowledge.However, the task is still limited to a restaurant information service, and has not yet been tested with a wide variety of dialog tasks. In addition, it is still unclear how to create intelligent dialog systems that can respond like a human agent. In consideration of these problems, we proposed a challenge track to the 6th dialog system technology challenges (DSTC6) using human-to-human dialog data to mimic human dialog behaviors. The focus of the challenge track is to train end-to-end conversation models from human-to-human conversation and accomplish end-to-end dialog tasks in various situations assuming a customer service, in which a system plays a role of human agent and generates natural and informative sentences in response to user's questions or comments given dialog context.

Paper Structure

This paper contains 19 sections, 1 equation, 2 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Tasks
  3. Data Collection
  4. Twitter data
  5. OpenSubtitles data
  6. Text Preprocessing
  7. Twitter data
  8. OpenSubtitles data
  9. Submitted Systems
  10. Evaluation
  11. Objective evaluation
  12. Subjective evaluation
  13. Results
  14. Summary
  15. Appendix
  16. ...and 4 more sections

Figures (2)

  • Figure 1: Sentence generation and evaluation in the end-to-end conversation modeling track.
  • Figure 2: Mean and standard deviation of human rating score.