AEQ-Bench: Measuring Empathy of Omni-Modal Large Models

TL;DR

AEQ-Bench introduces a multimodal empathy benchmark for omni-modal large models, pairing audio utterances with textual contexts to evaluate both empathetic response generation and empathy judgment from audio. It employs context- and tone-variant settings to probe linguistic and paralinguistic empathy, using 1,885 instances sourced from Meld, GigaSpeech, and EmoV-DB. Key findings show that audio-output models outperform text-only variants and that while models align with human judgments on coarse metrics, they struggle with fine-grained paralinguistic expressiveness, and textual captions cannot substitute for direct audio evaluation. The benchmark emphasizes the importance of audio-native evaluators for advancing genuinely empathetic AI, while also acknowledging ethical considerations and dataset limitations.

Abstract

While the automatic evaluation of omni-modal large models (OLMs) is essential, assessing empathy remains a significant challenge due to its inherent affectivity. To investigate this challenge, we introduce AEQ-Bench (Audio Empathy Quotient Benchmark), a novel benchmark to systematically assess two core empathetic capabilities of OLMs: (i) generating empathetic responses by comprehending affective cues from multi-modal inputs (audio + text), and (ii) judging the empathy of audio responses without relying on text transcription. Compared to existing benchmarks, AEQ-Bench incorporates two novel settings that vary in context specificity and speech tone. Comprehensive assessment across linguistic and paralinguistic metrics reveals that (1) OLMs trained with audio output capabilities generally outperformed models with text-only outputs, and (2) while OLMs align with human judgments for coarse-grained quality assessment, they remain unreliable for evaluating fine-grained paralinguistic expressiveness.

Figures (6)

• Figure 1: Overview of omni-modal tasks: instruction following (a & b) and conversation (c & d). The input configurations are: (a) audio content with textual instruction; (b) audio instruction with text, audio, or image content; (c) unimodal dialogue; and (d) mixed-modality dialogue (demonstrated by AEQ-Bench on the Meld subset). In the mixed-modality setting, the text provides context (e.g., chat history summary) for the current audio utterance, requiring the model to respond accordingly. Outputs across all tasks may be text, audio, or both.
• Figure 2: The constructed GigaSpeech subset of AEQ-Bench featuring context variation. For each utterance, we construct two plausible contexts, each associated with a corresponding reference response. (Appx. \ref{['sec:appendix:figure']})
• Figure 3: The constructed Emovdb subset of AEQ-Bench with tone variation. The middle part is the constructed context and the original audio. The utterances with different emotions/tones share the same context. Each audio tone (amused, disgusted, angry, and neutral) has a plausible explanation for its context and a corresponding response.
• Figure 4: Average 5-point paralinguistic (delivery) evaluation (y-axis) for model on x-axis. Higher is better. For judges, stars indicate human evaluation, while circles indicate OLM-judges using direct audio. Empty and filled triangles denote OLM-judges using additional contexts of the original and objective audio caption, respectively.
• Figure 5: The example of context-variant responses generated by GPT-4o. Its responses are typically long and adhere to a standardised structural format: first, a situation analysis, then suggestions, and finally a question.