Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark

Hiromi Wakaki, Yuki Mitsufuji, Yoshinori Maeda, Yukiko Nishimura, Silin Gao, Mengjie Zhao, Keiichi Yamada, Antoine Bosselut

TL;DR

ComperDial introduces a comprehensive benchmark for automatic dialogue evaluation by collecting diverse, persona-grounded conversations and multiple per-turn responses with human scores. It foregrounds context-aware, turn- and dialogue-level annotations to enable robust training and evaluation of metrics, and it pairs this with CPDScore, an LLM-based evaluator that offers auditable multi-turn assessments via Chain-of-Thought reasoning. The study demonstrates that CPDScore achieves higher alignment with human judgments than existing metrics on ComperDial and the USR benchmarks, with GPT-4 generally outperforming GPT-3.5 and two-step dialogue-level evaluation providing added benefits in certain settings. By releasing both ComperDial and CPDScore, the work aims to accelerate the development of robust, scalable automatic evaluation methods for open-domain dialogue systems and fosters broader, more reliable benchmarking in the field.

Abstract

We propose a new benchmark, ComperDial, which facilitates the training and evaluation of evaluation metrics for open-domain dialogue systems. ComperDial consists of human-scored responses for 10,395 dialogue turns in 1,485 conversations collected from 99 dialogue agents submitted to the Commonsense Persona-grounded Dialogue (CPD) challenge. As a result, for any dialogue, our benchmark includes multiple diverse responses with variety of characteristics to ensure more robust evaluation of learned dialogue metrics. In addition to single-turn response scores, ComperDial also contains dialogue-level human-annotated scores, enabling joint assessment of multi-turn model responses throughout a dialogue. Finally, building off ComperDial, we devise a new automatic evaluation metric to measure the general similarity of model-generated dialogues to human conversations. Our experimental results demonstrate that our novel metric, CPDScore is more correlated with human judgments than existing metrics. We release both ComperDial and CPDScore to the community to accelerate development of automatic evaluation metrics for open-domain dialogue systems.

ComperDial: Commonsense Persona-grounded Dialogue Dataset and Benchmark

TL;DR

ComperDial introduces a comprehensive benchmark for automatic dialogue evaluation by collecting diverse, persona-grounded conversations and multiple per-turn responses with human scores. It foregrounds context-aware, turn- and dialogue-level annotations to enable robust training and evaluation of metrics, and it pairs this with CPDScore, an LLM-based evaluator that offers auditable multi-turn assessments via Chain-of-Thought reasoning. The study demonstrates that CPDScore achieves higher alignment with human judgments than existing metrics on ComperDial and the USR benchmarks, with GPT-4 generally outperforming GPT-3.5 and two-step dialogue-level evaluation providing added benefits in certain settings. By releasing both ComperDial and CPDScore, the work aims to accelerate the development of robust, scalable automatic evaluation methods for open-domain dialogue systems and fosters broader, more reliable benchmarking in the field.

Abstract

We propose a new benchmark, ComperDial, which facilitates the training and evaluation of evaluation metrics for open-domain dialogue systems. ComperDial consists of human-scored responses for 10,395 dialogue turns in 1,485 conversations collected from 99 dialogue agents submitted to the Commonsense Persona-grounded Dialogue (CPD) challenge. As a result, for any dialogue, our benchmark includes multiple diverse responses with variety of characteristics to ensure more robust evaluation of learned dialogue metrics. In addition to single-turn response scores, ComperDial also contains dialogue-level human-annotated scores, enabling joint assessment of multi-turn model responses throughout a dialogue. Finally, building off ComperDial, we devise a new automatic evaluation metric to measure the general similarity of model-generated dialogues to human conversations. Our experimental results demonstrate that our novel metric, CPDScore is more correlated with human judgments than existing metrics. We release both ComperDial and CPDScore to the community to accelerate development of automatic evaluation metrics for open-domain dialogue systems.
Paper Structure (58 sections, 2 figures, 16 tables)

This paper contains 58 sections, 2 figures, 16 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Automatic dialogue evaluation
  4. Human evaluation
  5. Evaluation techniques
  6. Dialogue evaluation benchmarks
  7. ComperDial
  8. Persona profile creation
  9. Diversified dialogue collection
  10. Human dialogue collection
  11. Diversified dialogue response extension
  12. Human evaluation
  13. CPDScore
  14. Benchmark data analysis
  15. Response collection
  16. ...and 43 more sections

Figures (2)

  • Figure 1: Data collection pipeline of ComperDial.
  • Figure 2: Dialogue evaluation techniques.(a) Static single-turn evaluation(b) Interactive multi-turn / dialogue-level evaluation(c) Static multi-turn / dialogue-level evaluation (ours)