Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DialogBench: Evaluating LLMs as Human-like Dialogue Systems

Jiao Ou, Junda Lu, Che Liu, Yihong Tang, Fuzheng Zhang, Di Zhang, Kun Gai

TL;DR

DialogBench introduces a comprehensive benchmark to evaluate LLMs as human-like dialogue systems across 12 tasks covering coherence, consistency, correctness, and safety. It leverages GPT-4 as a data generator with bias mitigation and a filtering pipeline to create new, leakage-free evaluation instances in English and Chinese, enabling fair comparisons across 26 LLMs (pretrained and instruction-tuned). The study finds that instruction tuning yields improvements in human-likeness but reveals substantial gaps, particularly in emotion perception and daily-life knowledge, with GPT-4 remaining the strongest exemplar. The work provides a reproducible framework and insights that guide future development toward more human-like, emotionally aware, and contextually grounded dialogue agents, while acknowledging limitations in multilingual reach and closed-model reproducibility.

Abstract

Large language models (LLMs) have achieved remarkable breakthroughs in new dialogue capabilities by leveraging instruction tuning, which refreshes human impressions of dialogue systems. The long-standing goal of dialogue systems is to be human-like enough to establish long-term connections with users. Therefore, there has been an urgent need to evaluate LLMs as human-like dialogue systems. In this paper, we propose DialogBench, a dialogue evaluation benchmark that contains 12 dialogue tasks to probe the capabilities of LLMs as human-like dialogue systems should have. Specifically, we prompt GPT-4 to generate evaluation instances for each task. We first design the basic prompt based on widely used design principles and further mitigate the existing biases to generate higher-quality evaluation instances. Our extensive tests on English and Chinese DialogBench of 26 LLMs show that instruction tuning improves the human likeness of LLMs to a certain extent, but most LLMs still have much room for improvement as human-like dialogue systems. Interestingly, results also show that the positioning of assistant AI can make instruction tuning weaken the human emotional perception of LLMs and their mastery of information about human daily life.

DialogBench: Evaluating LLMs as Human-like Dialogue Systems

TL;DR

DialogBench introduces a comprehensive benchmark to evaluate LLMs as human-like dialogue systems across 12 tasks covering coherence, consistency, correctness, and safety. It leverages GPT-4 as a data generator with bias mitigation and a filtering pipeline to create new, leakage-free evaluation instances in English and Chinese, enabling fair comparisons across 26 LLMs (pretrained and instruction-tuned). The study finds that instruction tuning yields improvements in human-likeness but reveals substantial gaps, particularly in emotion perception and daily-life knowledge, with GPT-4 remaining the strongest exemplar. The work provides a reproducible framework and insights that guide future development toward more human-like, emotionally aware, and contextually grounded dialogue agents, while acknowledging limitations in multilingual reach and closed-model reproducibility.

Abstract

Large language models (LLMs) have achieved remarkable breakthroughs in new dialogue capabilities by leveraging instruction tuning, which refreshes human impressions of dialogue systems. The long-standing goal of dialogue systems is to be human-like enough to establish long-term connections with users. Therefore, there has been an urgent need to evaluate LLMs as human-like dialogue systems. In this paper, we propose DialogBench, a dialogue evaluation benchmark that contains 12 dialogue tasks to probe the capabilities of LLMs as human-like dialogue systems should have. Specifically, we prompt GPT-4 to generate evaluation instances for each task. We first design the basic prompt based on widely used design principles and further mitigate the existing biases to generate higher-quality evaluation instances. Our extensive tests on English and Chinese DialogBench of 26 LLMs show that instruction tuning improves the human likeness of LLMs to a certain extent, but most LLMs still have much room for improvement as human-like dialogue systems. Interestingly, results also show that the positioning of assistant AI can make instruction tuning weaken the human emotional perception of LLMs and their mastery of information about human daily life.
Paper Structure (43 sections, 10 figures, 21 tables)

This paper contains 43 sections, 10 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation of LLMs.
  4. Dialogue Benchmarks.
  5. LLMs for Data Generation.
  6. DialogBench
  7. Task Selection
  8. Question Setting
  9. Prompt Formatting
  10. Quality Control
  11. Bias Mitigation.
  12. Data Filter.
  13. Experimental Setup
  14. Dataset Statistics.
  15. LLMs to Evaluate.
  16. ...and 28 more sections

Figures (10)

  • Figure 1: The overall architecture of DialogBench construction.
  • Figure 2: Task selection in DialogBench.
  • Figure 3: The basic prompt on the left and the external description for mitigating bias on the right. We take slot filling as an example. DOMAIN is the placeholder of the given domain. [DOMAIN_PROMPT] and [STYLE_PROMPT] are the positions of the corresponding description to add.
  • Figure 4: The template for multi-choice questions in DialogBench.The red text is the explanation.
  • Figure 5: (a) The proportion of each dialogue style on both speakers in all generated dialogues via the basic prompt; (b) the proportion via the optimized prompt. The inner ring is the proportion of speaker1's style, while the outer ring is the proportion of the corresponding speaker2's style given speaker1's style.
  • ...and 5 more figures