Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations

Shuai Huang, Wenxuan Zhao, Jun Gao

TL;DR

SI-Bench presents a socially grounded benchmark for large language models by leveraging 2,221 authentic human-to-human dialogues from a Chinese social network and a focused 312-dialogue, 8-model annotation subset. It introduces a process–outcome decoupled evaluation framework grounded in social information processing theory, enabling fine-grained assessment of contextual understanding, response strategy, and reply generation. The study reveals a pronounced thought–action gap: state-of-the-art models excel at reasoning about social contexts yet lag in producing high-quality final replies, and Chain-of-Thought prompts often degrade reply quality in social dialogues. These findings underscore the need to align internal reasoning with external conversational behavior and motivate refining CoT approaches for socially intelligent AI, with SI-Bench providing open data and a reproducible evaluation protocol to advance this goal.

Abstract

As large language models (LLMs) develop anthropomorphic abilities, they are increasingly being deployed as autonomous agents to interact with humans. However, evaluating their performance in realistic and complex social interactions remains a significant challenge. Most previous research built datasets through simulated agent-to-agent interactions, which fails to capture the authentic linguistic styles and relational dynamics found in real human conversations. To address this gap, we introduce SI-Bench, a novel benchmark designed to evaluate aspects of social intelligence in LLMs. Grounded in broad social science theories, SI-Bench contains 2,221 authentic multi-turn dialogues collected from a social networking application. We further selected a subset of 312 dialogues for manual annotation across 8 major models. The experiments show that SOTA models have surpassed the human expert in process reasoning under complex social situations, yet they still fall behind humans in reply quality. Moreover, introducing Chain-of-Thought (CoT) reasoning may degrade the performance of LLMs in social dialogue tasks. All datasets are openly available at https://github.com/SI-Bench/SI-Bench.git.

SI-Bench: Benchmarking Social Intelligence of Large Language Models in Human-to-Human Conversations

TL;DR

SI-Bench presents a socially grounded benchmark for large language models by leveraging 2,221 authentic human-to-human dialogues from a Chinese social network and a focused 312-dialogue, 8-model annotation subset. It introduces a process–outcome decoupled evaluation framework grounded in social information processing theory, enabling fine-grained assessment of contextual understanding, response strategy, and reply generation. The study reveals a pronounced thought–action gap: state-of-the-art models excel at reasoning about social contexts yet lag in producing high-quality final replies, and Chain-of-Thought prompts often degrade reply quality in social dialogues. These findings underscore the need to align internal reasoning with external conversational behavior and motivate refining CoT approaches for socially intelligent AI, with SI-Bench providing open data and a reproducible evaluation protocol to advance this goal.

Abstract

As large language models (LLMs) develop anthropomorphic abilities, they are increasingly being deployed as autonomous agents to interact with humans. However, evaluating their performance in realistic and complex social interactions remains a significant challenge. Most previous research built datasets through simulated agent-to-agent interactions, which fails to capture the authentic linguistic styles and relational dynamics found in real human conversations. To address this gap, we introduce SI-Bench, a novel benchmark designed to evaluate aspects of social intelligence in LLMs. Grounded in broad social science theories, SI-Bench contains 2,221 authentic multi-turn dialogues collected from a social networking application. We further selected a subset of 312 dialogues for manual annotation across 8 major models. The experiments show that SOTA models have surpassed the human expert in process reasoning under complex social situations, yet they still fall behind humans in reply quality. Moreover, introducing Chain-of-Thought (CoT) reasoning may degrade the performance of LLMs in social dialogue tasks. All datasets are openly available at https://github.com/SI-Bench/SI-Bench.git.
Paper Structure (34 sections, 7 equations, 9 figures, 8 tables)

This paper contains 34 sections, 7 equations, 9 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Benchmark Construction
  3. Data Source
  4. A Taxonomy of Complex Social Situations
  5. Tension I: Communication Challenges under Semantic Uncertainty
  6. Tension II: Social Challenges under Power Dynamics
  7. Data Collection
  8. Evaluation Framework
  9. Theoretical Foundation
  10. Contextual Understanding
  11. Response Strategy
  12. Reply Generation
  13. Overall Assessment
  14. Evaluation Rubric
  15. Evaluation Process
  16. ...and 19 more sections

Figures (9)

  • Figure 1: Overview of the SI-Bench framework and a sample model response.
  • Figure 2: Data Collection and Scoring Pipeline.
  • Figure 3: Distribution of Complex Situations in the Experimental Dataset.
  • Figure 4: Content richness and output length analysis of LLMs and the human expert.
  • Figure 5: CoT reply shows higher similarity with the CoT process than direct reply.
  • ...and 4 more figures