Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models

Li Zhou, Lutong Yu, You Lyu, Yihang Lin, Zefeng Zhao, Junyi Ao, Yuhao Zhang, Benyou Wang, Haizhou Li

TL;DR

EchoMind introduces a cohesive, interrelated multi-level benchmark to evaluate empathetic speech-language models by modeling the cognitive stages of understanding, vocal-cue perception, reasoning, and empathetic response generation. It standardizes 39 vocal attributes across 3 coarse and 12 fine-grained dimensions, with semantically neutral scripts and controlled vocal variations to isolate the impact of vocal delivery. The benchmark combines three task levels (understanding, reasoning, conversation) with diverse objective and subjective metrics, and it evaluates a diverse set of 12 advanced SLMs, revealing a persistent gap in leveraging vocal cues for emotionally aligned dialogue. Findings show strong content understanding but limited empathetic performance under expressive vocal cues, highlighting the need for models that tightly integrate linguistic content with paralinguistic signals to achieve truly human-like conversational empathy.

Abstract

Speech Language Models (SLMs) have made significant progress in spoken language understanding. Yet it remains unclear whether they can fully perceive non lexical vocal cues alongside spoken words, and respond with empathy that aligns with both emotional and contextual factors. Existing benchmarks typically evaluate linguistic, acoustic, reasoning, or dialogue abilities in isolation, overlooking the integration of these skills that is crucial for human-like, emotionally intelligent conversation. We present EchoMind, the first interrelated, multi-level benchmark that simulates the cognitive process of empathetic dialogue through sequential, context-linked tasks: spoken-content understanding, vocal-cue perception, integrated reasoning, and response generation. All tasks share identical and semantically neutral scripts that are free of explicit emotional or contextual cues, and controlled variations in vocal style are used to test the effect of delivery independent of the transcript. EchoMind is grounded in an empathy-oriented framework spanning 3 coarse and 12 fine-grained dimensions, encompassing 39 vocal attributes, and evaluated using both objective and subjective metrics. Testing 12 advanced SLMs reveals that even state-of-the-art models struggle with high-expressive vocal cues, limiting empathetic response quality. Analyses of prompt strength, speech source, and ideal vocal cue recognition reveal persistent weaknesses in instruction-following, resilience to natural speech variability, and effective use of vocal cues for empathy. These results underscore the need for SLMs that integrate linguistic content with diverse vocal cues to achieve truly empathetic conversational ability.

EchoMind: An Interrelated Multi-level Benchmark for Evaluating Empathetic Speech Language Models

TL;DR

EchoMind introduces a cohesive, interrelated multi-level benchmark to evaluate empathetic speech-language models by modeling the cognitive stages of understanding, vocal-cue perception, reasoning, and empathetic response generation. It standardizes 39 vocal attributes across 3 coarse and 12 fine-grained dimensions, with semantically neutral scripts and controlled vocal variations to isolate the impact of vocal delivery. The benchmark combines three task levels (understanding, reasoning, conversation) with diverse objective and subjective metrics, and it evaluates a diverse set of 12 advanced SLMs, revealing a persistent gap in leveraging vocal cues for emotionally aligned dialogue. Findings show strong content understanding but limited empathetic performance under expressive vocal cues, highlighting the need for models that tightly integrate linguistic content with paralinguistic signals to achieve truly human-like conversational empathy.

Abstract

Speech Language Models (SLMs) have made significant progress in spoken language understanding. Yet it remains unclear whether they can fully perceive non lexical vocal cues alongside spoken words, and respond with empathy that aligns with both emotional and contextual factors. Existing benchmarks typically evaluate linguistic, acoustic, reasoning, or dialogue abilities in isolation, overlooking the integration of these skills that is crucial for human-like, emotionally intelligent conversation. We present EchoMind, the first interrelated, multi-level benchmark that simulates the cognitive process of empathetic dialogue through sequential, context-linked tasks: spoken-content understanding, vocal-cue perception, integrated reasoning, and response generation. All tasks share identical and semantically neutral scripts that are free of explicit emotional or contextual cues, and controlled variations in vocal style are used to test the effect of delivery independent of the transcript. EchoMind is grounded in an empathy-oriented framework spanning 3 coarse and 12 fine-grained dimensions, encompassing 39 vocal attributes, and evaluated using both objective and subjective metrics. Testing 12 advanced SLMs reveals that even state-of-the-art models struggle with high-expressive vocal cues, limiting empathetic response quality. Analyses of prompt strength, speech source, and ideal vocal cue recognition reveal persistent weaknesses in instruction-following, resilience to natural speech variability, and effective use of vocal cues for empathy. These results underscore the need for SLMs that integrate linguistic content with diverse vocal cues to achieve truly empathetic conversational ability.
Paper Structure (29 sections, 4 figures, 14 tables)

This paper contains 29 sections, 4 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. EchoMind Benchmark Design
  4. Overview Of EchoMind Benchmark
  5. Audio Dataset Construction
  6. Multi-Level Tasks Formulation
  7. Evaluation Metrics
  8. Experiments
  9. Experimental Setup
  10. Evaluated SLMs.
  11. Prompts Setup.
  12. Audio Inputs Setup.
  13. Experimental Results
  14. Overall Performance -- The Vocal‑Cue Gap in Emotionally Intelligent Dialogue.
  15. Task Correlations – General Positive Association in Vocal‑Cue‑Aware Performance.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: The EchoMind framework & examples. (a) Multi‑level cognitive process simulation for empathetic dialogue: Level 1—Understanding through content (ASR) and voice (MCQs); Level 2—Reasoning by integrating content and voice (MCQs); Level 3—Conversation with contextually and emotionally aligned responses (Open-domain Response). (b) Responses under controlled vocal-style variations of the same script—target, neutral, and alternative expressions—illustrating differences in response focus.
  • Figure 2: Correlations between model performance in vocal‑cue‑aware understanding, reasoning, and conversational response quality (C4, VES; plus C1 in the right plot).
  • Figure 3: Sensitivity of conversational responses under three prompt settings—P1: zero-prompt, P2: basic, and P3: enhanced.
  • Figure 4: Performance differences (Human = recorded, TTS = synthesized) on EchoMind-Human scripts.