AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios

TL;DR

AgentSense introduces a bottom-up social intelligence benchmark for language-model–driven agents by deriving 1,225 interactive scenarios from real-world scripts. It uses Dramaturgical Theory and ERG-based goals to craft diverse, multi-agent settings with private information, evaluated through multi-turn interactions and third-party judgments. The framework includes a rigorous data-validation pipeline and leakage-mitigation strategies, and its experiments reveal that current models struggle with complex social goals and private-information reasoning, with performance varying across profiles and partner models. The work provides extensive data and code to promote reproducibility and further development of socially competent language agents.

Abstract

Large language models (LLMs) are increasingly leveraged to empower autonomous agents to simulate human beings in various fields of behavioral research. However, evaluating their capacity to navigate complex social interactions remains a challenge. Previous studies face limitations due to insufficient scenario diversity, complexity, and a single-perspective focus. To this end, we introduce AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarios. Drawing on Dramaturgical Theory, AgentSense employs a bottom-up approach to create 1,225 diverse social scenarios constructed from extensive scripts. We evaluate LLM-driven agents through multi-turn interactions, emphasizing both goal completion and implicit reasoning. We analyze goals using ERG theory and conduct comprehensive experiments. Our findings highlight that LLMs struggle with goals in complex social scenarios, especially high-level growth needs, and even GPT-4o requires improvement in private information reasoning. Code and data are available at \url{https://github.com/ljcleo/agent_sense}.

Figures (10)

• Figure 1: An illustration of challenging yet realistic social scenarios --- a family gathering and an office conversation, where the characters are driven by ChatGPT. While the dialogue could flow smoothly, Emily is unable to achieve her goals during the family gathering and fails to deduce Jordan's thoughts in an office setting.
• Figure 2: Overall framework of AgentSense. We construct scenario templates from scripts and synthesize characters to diversify the scenarios. Then, language models role-play the characters to interact with each other. After that, the participants and third-party judges are interviewed for evaluation.
• Figure 3: Scenario template construction pipeline (automated with Python and GPT-4o): (A) Scenario Extraction: We split the script into scenes then scenarios (1), and summarize their background and description (2), which are merged into a descriptive background for independent role-play (3). (B) Social Goal Extraction: We extract each character's social goals (4) and amend them by regenerating the whole scenario (5) and rewriting/deleting invalid goals (6). (C) Private Information Extraction: We determine if the scene involves private information inference (7); if yes, we extract private information as QA pairs (8) and generate private info records (9) and evaluation questions (10). (D) Leakage Mitigation and Template Generation: We remove elements associated with specific episodes and replace characters with slots for synthesized agents with similar characteristics to fill in (11).
• Figure 4: (a) Number of scenarios aligned with the eight categories under ERG theory. Each scenario may encompass multiple goals. (b) Moral values distribution of the agents. An individual may have multiple moral values, with those appearing fewer than 30 times categorized as Others. (c) Distribution of the agents' Big Five personality traits.
• Figure 5: (a) Judge majority score of interactions among different model-driven agents, highlighting that being a sender is more challenging. (b) Model performance as both attacker and defender, with notably weaker and less consistent results when acting as a defender.