Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Repository of Conversational Datasets

Matthew Henderson, Paweł Budzianowski, Iñigo Casanueva, Sam Coope, Daniela Gerz, Girish Kumar, Nikola Mrkšić, Georgios Spithourakis, Pei-Hao Su, Ivan Vulić, Tsung-Hsien Wen

TL;DR

The paper introduces a public repository of large, diverse conversational datasets (Reddit, OpenSubtitles, AmazonQA) with a standardized response-selection evaluation framework using 1-of-100 accuracy. It provides reproducible data processing scripts and benchmarks across keyword-based, vector-based, and neural encoder models, including a strong, fully-trained polyai-encoder. Key findings show the encoder outperforms baselines, with tf-idf/BM25 remaining competitive in some domains, and USE surpassing ELMo, highlighting the value of large-scale, diverse data for pretraining and downstream conversation tasks. This work offers a practical, scalable resource to accelerate research and benchmarking in conversational AI, while inviting community contributions for expansion and refinement.

Abstract

Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches. To this end, we present a repository of conversational datasets consisting of hundreds of millions of examples, and a standardised evaluation procedure for conversational response selection models using '1-of-100 accuracy'. The repository contains scripts that allow researchers to reproduce the standard datasets, or to adapt the pre-processing and data filtering steps to their needs. We introduce and evaluate several competitive baselines for conversational response selection, whose implementations are shared in the repository, as well as a neural encoder model that is trained on the entire training set.

A Repository of Conversational Datasets

TL;DR

The paper introduces a public repository of large, diverse conversational datasets (Reddit, OpenSubtitles, AmazonQA) with a standardized response-selection evaluation framework using 1-of-100 accuracy. It provides reproducible data processing scripts and benchmarks across keyword-based, vector-based, and neural encoder models, including a strong, fully-trained polyai-encoder. Key findings show the encoder outperforms baselines, with tf-idf/BM25 remaining competitive in some domains, and USE surpassing ELMo, highlighting the value of large-scale, diverse data for pretraining and downstream conversation tasks. This work offers a practical, scalable resource to accelerate research and benchmarking in conversational AI, while inviting community contributions for expansion and refinement.

Abstract

Progress in Machine Learning is often driven by the availability of large datasets, and consistent evaluation metrics for comparing modeling approaches. To this end, we present a repository of conversational datasets consisting of hundreds of millions of examples, and a standardised evaluation procedure for conversational response selection models using '1-of-100 accuracy'. The repository contains scripts that allow researchers to reproduce the standard datasets, or to adapt the pre-processing and data filtering steps to their needs. We introduce and evaluate several competitive baselines for conversational response selection, whose implementations are shared in the repository, as well as a neural encoder model that is trained on the entire training set.

Paper Structure

This paper contains 13 sections, 1 equation, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Conversational Dataset Format
  3. Datasets
  4. Reddit
  5. OpenSubtitles
  6. AmazonQA
  7. Response Selection Task
  8. Keyword-based Methods
  9. Vector-based Methods
  10. Encoder Model
  11. Evaluation
  12. Conclusion
  13. Appendix

Figures (3)

  • Figure 1: An illustrative Tensorflow example in a conversational dataset, consisting of a conversational context and an appropriate response. Each string is stored as a bytes feature using its UTF-8 encoding.
  • Figure 2: Two examples illustrating the conversational response selection task: given the input context sentence, the goal is to identify the relevant response from a large pool of candidate responses.
  • Figure 3: Examples from the three datasets. Each example is a mapping from feature names to string features. Features with a star $\star$ are used to compute the deterministic train/test split.