MULTI-Bench: A Multi-Turn Interactive Benchmark for Assessing Emotional Intelligence ability of Spoken Dialogue Models

TL;DR

Multi-Bench addresses the gap in evaluating emotional intelligence of Spoken Dialogue Models (SDMs) within genuinely interactive, multi-turn contexts. It introduces a hierarchical two-track framework with five tasks and about 3,212 samples, plus a reproducible end-to-end evaluation loop that integrates audio and text judgements along with paralinguistic cues. Experimental results show GPT-4o achieving the best overall performance, though emotion recognition and sustained emotionally intelligent dialogue remain challenging for most models, especially in English. By providing a structured, audial-textual evaluation pipeline and diverse data sources, Multi-Bench offers a practical resource to drive development of emotionally intelligent SDMs and to guide future research on both linguistic and acoustic aspects of EI.

Abstract

Spoken Dialogue Models (SDMs) have advanced rapidly, yet their ability to sustain genuinely interactive multi-turn conversations remains underexplored, as most benchmarks focus on single-turn exchanges. We introduce Multi-Bench, the first benchmark explicitly designed to evaluate SDMs in multi-turn interactive dialogue with an emphasis on emotional intelligence. Multi-Bench employs a hierarchical structure with a basic track for emotion understanding and reasoning and an advanced track for emotion support and application. It comprises five carefully designed tasks and about 3.2K samples, ranging from emotion recognition to complex reasoning and interactive dialogue, supported by a reproducible evaluation framework. We evaluate six representative SDMs on eight subsets of Multi-Bench. Results show that while current SDMs achieve good performance on basic understanding tasks, they still have room for improvement in advanced multi-turn interactive dialogue and reasoning-related tasks, particularly in emotion awareness and application.

Figures (2)

• Figure 1: Illustration of the proposed multi-turn interactive evaluation framework in Multi-Bench.
• Figure 2: Example data for the five sub-tasks in Multi-Bench: Emotion Recognition, Paralinguistic Recognition, Emotion Inference, Style Inference, and Interactive Dialogue.