A Large Collection of Model-generated Contradictory Responses for Consistency-aware Dialogue Systems

TL;DR

A large dataset of response generation models' contradictions for the first time is built and valuable insights into the characteristics of model-generated contradictions are acquired through an extensive analysis of the collected responses.

Abstract

Mitigating the generation of contradictory responses poses a substantial challenge in dialogue response generation. The quality and quantity of available contradictory response data play a vital role in suppressing these contradictions, offering two significant benefits. First, having access to large contradiction data enables a comprehensive examination of their characteristics. Second, data-driven methods to mitigate contradictions may be enhanced with large-scale contradiction data for training. Nevertheless, no attempt has been made to build an extensive collection of model-generated contradictory responses. In this paper, we build a large dataset of response generation models' contradictions for the first time. Then, we acquire valuable insights into the characteristics of model-generated contradictions through an extensive analysis of the collected responses. Lastly, we also demonstrate how this dataset substantially enhances the performance of data-driven contradiction suppression methods.

Figures (2)

• Figure 1: Overview of our data collection process.
• Figure 2: Example of $C_{\mathrm{w/o}\{u_q, C_{\mathrm{mid}}\}}$ and $C_{\mathrm{w/o}\{u_q, r\}}$.