Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems

Haifeng Jing, Yujie Hou, Junfei Liu, Rui Xie, alan Xu, Jinlong Ma, Qichun Deng

TL;DR

MoodBench 1.0 defines a formal notion of emotional companionship dialogue (ECD) and delivers the first purpose-built benchmark for ECD evaluation. The paper introduces a four-layer Ability→Task→Data→Method framework, decomposes ECD capabilities into Threshold, Foundational, and Core abilities, and constructs 41 tasks across 60 datasets with 60 evaluation methods to quantify emotional companionship across 30 models. Empirical results demonstrate MoodBench’s discriminant validity and alignment with human judgments, while revealing persistent gaps in long-term memory, personalization, and deep companionship, thereby providing a concrete roadmap for targeted improvements. The work enables developers to diagnose weaknesses, prioritize optimization, and steer future research toward multilingual, multimodal, and memory-enhanced ECDs with practical impact on user experience.

Abstract

With the rapid development of Large Language Models, dialogue systems are shifting from information tools to emotional companions, heralding the era of Emotional Companionship Dialogue Systems (ECDs) that provide personalized emotional support for users. However, the field lacks clear definitions and systematic evaluation standards for ECDs. To address this, we first propose a definition of ECDs with formal descriptions. Then, based on this theory and the design principle of "Ability Layer-Task Layer (three level)-Data Layer-Method Layer", we design and implement the first ECD evaluation benchmark - MoodBench 1.0. Through extensive evaluations of 30 mainstream models, we demonstrate that MoodBench 1.0 has excellent discriminant validity and can effectively quantify the differences in emotional companionship abilities among models. Furthermore, the results reveal current models' shortcomings in deep emotional companionship, guiding future technological optimization and significantly aiding developers in enhancing ECDs' user experience.

MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems

TL;DR

MoodBench 1.0 defines a formal notion of emotional companionship dialogue (ECD) and delivers the first purpose-built benchmark for ECD evaluation. The paper introduces a four-layer Ability→Task→Data→Method framework, decomposes ECD capabilities into Threshold, Foundational, and Core abilities, and constructs 41 tasks across 60 datasets with 60 evaluation methods to quantify emotional companionship across 30 models. Empirical results demonstrate MoodBench’s discriminant validity and alignment with human judgments, while revealing persistent gaps in long-term memory, personalization, and deep companionship, thereby providing a concrete roadmap for targeted improvements. The work enables developers to diagnose weaknesses, prioritize optimization, and steer future research toward multilingual, multimodal, and memory-enhanced ECDs with practical impact on user experience.

Abstract

With the rapid development of Large Language Models, dialogue systems are shifting from information tools to emotional companions, heralding the era of Emotional Companionship Dialogue Systems (ECDs) that provide personalized emotional support for users. However, the field lacks clear definitions and systematic evaluation standards for ECDs. To address this, we first propose a definition of ECDs with formal descriptions. Then, based on this theory and the design principle of "Ability Layer-Task Layer (three level)-Data Layer-Method Layer", we design and implement the first ECD evaluation benchmark - MoodBench 1.0. Through extensive evaluations of 30 mainstream models, we demonstrate that MoodBench 1.0 has excellent discriminant validity and can effectively quantify the differences in emotional companionship abilities among models. Furthermore, the results reveal current models' shortcomings in deep emotional companionship, guiding future technological optimization and significantly aiding developers in enhancing ECDs' user experience.

Paper Structure

This paper contains 31 sections, 2 equations, 7 figures, 1 table, 5 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Dialogue System
  4. Evaluation of dialogue system models
  5. Definition of ECD
  6. MoodBench 1.0
  7. Ability Decomposition Process
  8. Score Aggregation Process
  9. MoodBench 1.0 Implementation and Analysis
  10. Experiment
  11. Validation of Benchmark Effectiveness and Correctness
  12. patterns and trends in current ECDs
  13. Pinpointing Directions for ECD Optimization
  14. Conclusion
  15. Appendix
  16. ...and 16 more sections

Figures (7)

  • Figure 1: The relationship of TOD, ODD and ECD
  • Figure 2: The Four-Layer Evaluation Framework of MoodBench 1.0
  • Figure 3: Decomposition of Emotional Companionship Ability
  • Figure 4: Score Distribution
  • Figure 5: Average Score Comparison: Closed-source vs. Open-source Models
  • ...and 2 more figures