EmotionQueen: A Benchmark for Evaluating Empathy of Large Language Models

TL;DR

EmotionQueen addresses the need for a unified, objective benchmark to assess large language models' emotional intelligence beyond standard sentiment tasks. It defines four tasks—Key Event Recognition, Mixed Event Recognition, Implicit Emotional Recognition, and Intention Recognition—and two binary metrics (PASS and WIN) to evaluate recognition and empathetic response. The study builds a dataset of 10,000 user statements across five life scenarios and evaluates a broad set of LLMs, highlighting that LLaMA-70B and Claude2 achieve strong overall performance while revealing gaps in empathetic reasoning. Findings include a weak or absent correlation between PASS and WIN, and a high alignment between automatic and human judgments (Pearson 0.991). The work highlights practical implications for deploying emotionally aware AI and motivates future improvements in scope, fairness, and safety.

Abstract

Emotional intelligence in large language models (LLMs) is of great importance in Natural Language Processing. However, the previous research mainly focus on basic sentiment analysis tasks, such as emotion recognition, which is not enough to evaluate LLMs' overall emotional intelligence. Therefore, this paper presents a novel framework named EmotionQueen for evaluating the emotional intelligence of LLMs. The framework includes four distinctive tasks: Key Event Recognition, Mixed Event Recognition, Implicit Emotional Recognition, and Intention Recognition. LLMs are requested to recognize important event or implicit emotions and generate empathetic response. We also design two metrics to evaluate LLMs' capabilities in recognition and response for emotion-related statements. Experiments yield significant conclusions about LLMs' capabilities and limitations in emotion intelligence.

Figures (21)

• Figure 1: Responses with and without empathy in four real-world scenarios.
• Figure 2: The overview of the proposed EmotionQueen benchmark, including four tasks.
• Figure 3: The overall performance of different LLMs in the proposed EmotionQueen benchmark.
• Figure 4: The relationship between PASS rate and WIN rate of different LLMs in four tasks, respectively.
• Figure 5: The relationship between PASS rate and WIN rate of different LLMs in five categories of events, respectively.