How Social is It? A Benchmark for LLMs' Capabilities in Multi-user Multi-turn Social Agent Tasks

TL;DR

This work targets the gap in evaluating Large Language Models (LLMs) within real-world, multi-user, multi-turn social settings by introducing How Social Is It (HSII), a sociology-grounded benchmark and four-stage evaluation framework. It pairs a news-derived HSII-Dataset with an explicit agent-task leveling scheme and a novel HSII score, augmented by a Chain-of-Thought (COT) complexity metric to balance accuracy and efficiency. Across multiple LLMs and humans, the study shows GPT-4 generally achieves the best HSII performance, though humans still outperform models, and COT prompting yields measurable gains for several lagging models, while revealing COT complexity as a useful discriminative metric. The contributions provide a principled, scalable approach to benchmark social-agent capabilities and inform design of multi-user social AI systems, with future work aimed at expanding dataset scale and diversifying social scenarios. The HSII framework is mathematically underpinned by the score $\iota = r_1(1+\alpha r_2(1+\beta (r_3+\gamma r_4)))$, where $r_1$–$r_4$ are distinct evaluation stages and $\alpha,\beta,\gamma$ are tunable weights, allowing nuanced performance aggregation across parsing, targeting, and sustained dialogue tasks.$

Abstract

Expanding the application of large language models (LLMs) to societal life, instead of primary function only as auxiliary assistants to communicate with only one person at a time, necessitates LLMs' capabilities to independently play roles in multi-user, multi-turn social agent tasks within complex social settings. However, currently the capability has not been systematically measured with available benchmarks. To address this gap, we first introduce an agent task leveling framework grounded in sociological principles. Concurrently, we propose a novel benchmark, How Social Is It (we call it HSII below), designed to assess LLM's social capabilities in comprehensive social agents tasks and benchmark representative models. HSII comprises four stages: format parsing, target selection, target switching conversation, and stable conversation, which collectively evaluate the communication and task completion capabilities of LLMs within realistic social interaction scenarios dataset, HSII-Dataset. The dataset is derived step by step from news dataset. We perform an ablation study by doing clustering to the dataset. Additionally, we investigate the impact of chain of thought (COT) method on enhancing LLMs' social performance. Since COT cost more computation, we further introduce a new statistical metric, COT-complexity, to quantify the efficiency of certain LLMs with COTs for specific social tasks and strike a better trade-off between measurement of correctness and efficiency. Various results of our experiments demonstrate that our benchmark is well-suited for evaluating social skills in LLMs.

Figures (4)

• Figure 1: Main leveling of agent task and capability evaluation. On the left is different levels of social tasks including basic single-user tasks, multi-agents and multi-users tasks based on the first ones, and final multi-task ones on top. We mainly put sight on multi-user tasks. Then on the right is our four-step evaluation framework for multi-user tasks.
• Figure 2: Evaluation dataset construction design and HSII evaluation framework pipeline.
• Figure 3: Clustering analysis of constructed dataset. Each color stands for one cluster of HSII dataset, mainly matching one field or paradox feature in social scenes.
• Figure 4: One example in our COT set.

Theorems & Definitions (2)

• definition thmcounterdefinition
• definition thmcounterdefinition