DialogCC: An Automated Pipeline for Creating High-Quality Multi-Modal Dialogue Dataset

TL;DR

DialogCC introduces a fully automatic pipeline to construct a high-quality, diverse multi-modal dialogue dataset by leveraging GPT-4 to infer image-sharing moments and CLIP to align images with dialogue context. The pipeline collects data from five text-only dialogue corpora and CC3M, then aligns and filters to produce DialogCC, which exhibits superior data quality and diversity and improves generalization to unseen dialogue scenarios. Empirical results show that models trained on DialogCC achieve stronger cross-dataset performance and better comprehension of image–dialogue interactions than models trained on existing baselines, with scaling yielding additional gains. The work provides an end-to-end, reproducible framework and releases both code and data to facilitate further research in automated, high-quality multi-modal dialogue dataset construction.

Abstract

As sharing images in an instant message is a crucial factor, there has been active research on learning an image-text multi-modal dialogue models. However, training a well-generalized multi-modal dialogue model remains challenging due to the low quality and limited diversity of images per dialogue in existing multi-modal dialogue datasets. In this paper, we propose an automated pipeline to construct a multi-modal dialogue dataset, ensuring both dialogue quality and image diversity without requiring minimum human effort. In our pipeline, to guarantee the coherence between images and dialogue, we prompt GPT-4 to infer potential image-sharing moments - specifically, the utterance, speaker, rationale, and image description. Furthermore, we leverage CLIP similarity to maintain consistency between aligned multiple images to the utterance. Through this pipeline, we introduce DialogCC, a high-quality and diverse multi-modal dialogue dataset that surpasses existing datasets in terms of quality and diversity in human evaluation. Our comprehensive experiments highlight that when multi-modal dialogue models are trained using our dataset, their generalization performance on unseen dialogue datasets is significantly enhanced. We make our source code and dataset publicly available.

Figures (22)

• Figure 1: Comparing DialogCC (ours) to three existing multi-modal dialogue dataset in terms of a quality, diversity, and generalization. RC@1 denotes the averaged contributed R@1 performance.
• Figure 2: An overview of our proposed automatic pipeline for creating a high-quality and diverse multi-modal dialogue dataset.
• Figure 3: We show the examples of frequently matched images. The number under each image indicates the count of how many utterances are matched.
• Figure 4: We illustrate the inconsistent images filtering process.
• Figure 5: Results of head-to-head comparison between DialogCC (ours) and three existing datasets: PhotoChat, MMDialog, MMDD.