Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Academically intelligent LLMs are not necessarily socially intelligent

Ruoxi Xu, Hongyu Lin, Xianpei Han, Le Sun, Yingfei Sun

TL;DR

An extensive evaluation of the social intelligence of large language models with 13 recent popular and state-of-art LLM agents on SESI indicates the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors.

Abstract

The academic intelligence of large language models (LLMs) has made remarkable progress in recent times, but their social intelligence performance remains unclear. Inspired by established human social intelligence frameworks, particularly Daniel Goleman's social intelligence theory, we have developed a standardized social intelligence test based on real-world social scenarios to comprehensively assess the social intelligence of LLMs, termed as the Situational Evaluation of Social Intelligence (SESI). We conducted an extensive evaluation with 13 recent popular and state-of-art LLM agents on SESI. The results indicate the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors. Moreover, there exists a relatively low correlation between the social intelligence and academic intelligence exhibited by LLMs, suggesting that social intelligence is distinct from academic intelligence for LLMs. Additionally, while it is observed that LLMs can't ``understand'' what social intelligence is, their social intelligence, similar to that of humans, is influenced by social factors.

Academically intelligent LLMs are not necessarily socially intelligent

TL;DR

An extensive evaluation of the social intelligence of large language models with 13 recent popular and state-of-art LLM agents on SESI indicates the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors.

Abstract

The academic intelligence of large language models (LLMs) has made remarkable progress in recent times, but their social intelligence performance remains unclear. Inspired by established human social intelligence frameworks, particularly Daniel Goleman's social intelligence theory, we have developed a standardized social intelligence test based on real-world social scenarios to comprehensively assess the social intelligence of LLMs, termed as the Situational Evaluation of Social Intelligence (SESI). We conducted an extensive evaluation with 13 recent popular and state-of-art LLM agents on SESI. The results indicate the social intelligence of LLMs still has significant room for improvement, with superficially friendliness as a primary reason for errors. Moreover, there exists a relatively low correlation between the social intelligence and academic intelligence exhibited by LLMs, suggesting that social intelligence is distinct from academic intelligence for LLMs. Additionally, while it is observed that LLMs can't ``understand'' what social intelligence is, their social intelligence, similar to that of humans, is influenced by social factors.
Paper Structure (34 sections, 11 figures, 8 tables)

This paper contains 34 sections, 11 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Results
  3. For LLMs, social intelligence is distinct from academic intelligence
  4. LLMs are superficially friendly
  5. LLMs can't "understand" what social intelligence is
  6. Social intelligence of LLMs, similar to that of human beings, is influenced by social factors
  7. Personality, gender, role and person significantly influence social intelligence of LLM agents.
  8. LLM agents with extroverted but disagreeable personality consistently exhibit higher social intelligence.
  9. LLM agents with male gender generally exhibit higher social intelligence.
  10. The LLM agents with family and work roles generally exhibit higher social intelligence than with romatic and friend roles.
  11. The LLM agents with field perspective generally exhibit higher social intelligence than with observer perspective.
  12. SESI: The Situational Evaluation of Social Intelligence
  13. Introduction to SESI
  14. Social intelligence components in SESI
  15. The development of SESI
  16. ...and 19 more sections

Figures (11)

  • Figure 1: Overview of the situational evaluation of social intelligence. Green indicates the correct answer and red indicates the wrong answer selected by gpt-3.5-turbo-0613 model.
  • Figure 1: Proportions of error causes on SESI. The primary error causes include superficially friendly by making social judgements based on superficially friendly patterns, sidestepping question by providing irrelevant responses and excessively general by providing excessively generalized and unhelpful answers.
  • Figure 2: Heatmap for correlation matrix for social and academic intelligence measures. Intuitively, there is a comparatively low correlation between the performance of LLM agents in social intelligence and academic intelligence.
  • Figure 2: Social intelligence performance of LLM agents under varying levels of social intelligence prompts. From the figure, the actual social intelligence performance of LLM agents diverges from or even opposes the indicated levels of prompts, indicating a misconception of social intelligence by LLM agents.
  • Figure 3: Change Ratio in the social intelligence performance of LLM agents following the manipulation of factors. The significance of differences between each factor and the control prompt (no factor) is denoted by ns: $p > 0.05, \prescript{*}{}{p} < 0.05, \prescript{**}{}{p} < 0.01, \prescript{***}{}{p} < 0.001, \prescript{****}{}{p} < 0.0001$. As illustrated in the Figure, the impact of factors on the social intelligence performance of LLM agents is model-dependent. For at least one LLM agent, personality, gender, role and person exhibit significant effects on their social intelligence.
  • ...and 6 more figures