Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HeartBench: Probing Core Dimensions of Anthropomorphic Intelligence in LLMs

Jiaxin Liu, Peiyi Tu, Wenyu Chen, Yihong Zhuang, Xinxia Ling, Anji Zhou, Chenxi Wang, Zhuo Han, Zhengkai Yang, Junbo Zhao, Zenan Huang, Yuanyuan Wang

TL;DR

HeartBench introduces a theory-driven benchmark to evaluate anthropomorphic intelligence in Chinese LLMs, integrating emotional, cultural, and ethical dimensions via authentic counseling scenarios. The framework uses a 5×15 taxonomy, a 2,818-item rubric, and a reasoning-before-scoring protocol with an automated judge to convert qualitative judgments into binary criteria. Empirical results show a ~60% alignment with expert ideals across 13 models, with notable weaknesses in curiosity, autonomy, and ethical reasoning, especially in hard subtexts. A 30% double-blind validation yields 87% agreement with human experts, supporting HeartBench as a reliable tool for diagnosing socio-emotional AI gaps and guiding data/alignment efforts. The work demonstrates that anthropomorphic intelligence is a distinct capability requiring targeted, human-centric training and evaluation beyond standard cognitive benchmarks.

Abstract

While Large Language Models (LLMs) have achieved remarkable success in cognitive and reasoning benchmarks, they exhibit a persistent deficit in anthropomorphic intelligence-the capacity to navigate complex social, emotional, and ethical nuances. This gap is particularly acute in the Chinese linguistic and cultural context, where a lack of specialized evaluation frameworks and high-quality socio-emotional data impedes progress. To address these limitations, we present HeartBench, a framework designed to evaluate the integrated emotional, cultural, and ethical dimensions of Chinese LLMs. Grounded in authentic psychological counseling scenarios and developed in collaboration with clinical experts, the benchmark is structured around a theory-driven taxonomy comprising five primary dimensions and 15 secondary capabilities. We implement a case-specific, rubric-based methodology that translates abstract human-like traits into granular, measurable criteria through a ``reasoning-before-scoring'' evaluation protocol. Our assessment of 13 state-of-the-art LLMs indicates a substantial performance ceiling: even leading models achieve only 60% of the expert-defined ideal score. Furthermore, analysis using a difficulty-stratified ``Hard Set'' reveals a significant performance decay in scenarios involving subtle emotional subtexts and complex ethical trade-offs. HeartBench establishes a standardized metric for anthropomorphic AI evaluation and provides a methodological blueprint for constructing high-quality, human-aligned training data.

HeartBench: Probing Core Dimensions of Anthropomorphic Intelligence in LLMs

TL;DR

HeartBench introduces a theory-driven benchmark to evaluate anthropomorphic intelligence in Chinese LLMs, integrating emotional, cultural, and ethical dimensions via authentic counseling scenarios. The framework uses a 5×15 taxonomy, a 2,818-item rubric, and a reasoning-before-scoring protocol with an automated judge to convert qualitative judgments into binary criteria. Empirical results show a ~60% alignment with expert ideals across 13 models, with notable weaknesses in curiosity, autonomy, and ethical reasoning, especially in hard subtexts. A 30% double-blind validation yields 87% agreement with human experts, supporting HeartBench as a reliable tool for diagnosing socio-emotional AI gaps and guiding data/alignment efforts. The work demonstrates that anthropomorphic intelligence is a distinct capability requiring targeted, human-centric training and evaluation beyond standard cognitive benchmarks.

Abstract

While Large Language Models (LLMs) have achieved remarkable success in cognitive and reasoning benchmarks, they exhibit a persistent deficit in anthropomorphic intelligence-the capacity to navigate complex social, emotional, and ethical nuances. This gap is particularly acute in the Chinese linguistic and cultural context, where a lack of specialized evaluation frameworks and high-quality socio-emotional data impedes progress. To address these limitations, we present HeartBench, a framework designed to evaluate the integrated emotional, cultural, and ethical dimensions of Chinese LLMs. Grounded in authentic psychological counseling scenarios and developed in collaboration with clinical experts, the benchmark is structured around a theory-driven taxonomy comprising five primary dimensions and 15 secondary capabilities. We implement a case-specific, rubric-based methodology that translates abstract human-like traits into granular, measurable criteria through a ``reasoning-before-scoring'' evaluation protocol. Our assessment of 13 state-of-the-art LLMs indicates a substantial performance ceiling: even leading models achieve only 60% of the expert-defined ideal score. Furthermore, analysis using a difficulty-stratified ``Hard Set'' reveals a significant performance decay in scenarios involving subtle emotional subtexts and complex ethical trade-offs. HeartBench establishes a standardized metric for anthropomorphic AI evaluation and provides a methodological blueprint for constructing high-quality, human-aligned training data.
Paper Structure (36 sections, 1 equation, 5 figures, 1 table)

This paper contains 36 sections, 1 equation, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. HeartBench
  4. The HeartBench Taxonomy
  5. Conceptual Framework
  6. Scenario Design
  7. Difficulty Stratification
  8. Data Construction
  9. Seed Corpus Construction
  10. Human-in-the-Loop Curation
  11. Professional Alignment
  12. Data Refinement
  13. Case-specific Evaluation Criteria
  14. Expertise amortization
  15. Quality control
  16. ...and 21 more sections

Figures (5)

  • Figure 1: Rubric distribution
  • Figure 2: The construction and evaluation pipeline of HeartBench. The process begins by collecting diverse data to establish a conceptual framework and scenario design. LLMs are then employed to rewrite the content, ensuring privacy and logical consistency. Following expert review and iterative rubric generation, the HeartBench evaluation system utilizes difficulty stratification and LLM-as-a-Judge to provide comprehensive scoring for each case.
  • Figure 3: A HeartBench example consists of a conversation and expert-written rubric criteria specific to that conversation. A model-based grader scores the response against each criterion.
  • Figure 4: Overall Dimension Performance
  • Figure 5: Hard Set Dimension Performance