Exploring Prosocial Irrationality for LLM Agents: A Social Cognition View

TL;DR

The paper investigates whether LLM Agents' systematic hallucinations reflect human cognitive biases and can be harnessed to reveal and enhance social intelligence. It introduces CogMir, an extensible framework for evaluating social intelligence via cognitive-bias manipulations, with Mirror Environmental Settings, modular components, and seven bias subsets. Experimental results across seven LLMs show broad alignment with human pro-social biases (e.g., Herd, Authority, Ben Franklin, Halo, Confirmations) but also model-specific deviations (Rumor Chain, Gambler's Fallacy) and greater sensitivity to certainty and social status. Overall, CogMir provides interpretable tools and a platform for future cross-disciplinary research into LLM Agent social dynamics and ethics.

Abstract

Large language models (LLMs) have been shown to face hallucination issues due to the data they trained on often containing human bias; whether this is reflected in the decision-making process of LLM Agents remains under-explored. As LLM Agents are increasingly employed in intricate social environments, a pressing and natural question emerges: Can we utilize LLM Agents' systematic hallucinations to mirror human cognitive biases, thus exhibiting irrational social intelligence? In this paper, we probe the irrational behavior among contemporary LLM Agents by melding practical social science experiments with theoretical insights. Specifically, We propose CogMir, an open-ended Multi-LLM Agents framework that utilizes hallucination properties to assess and enhance LLM Agents' social intelligence through cognitive biases. Experimental results on CogMir subsets show that LLM Agents and humans exhibit high consistency in irrational and prosocial decision-making under uncertain conditions, underscoring the prosociality of LLM Agents as social entities and highlighting the significance of hallucination properties. Additionally, the CogMir framework demonstrates its potential as a valuable platform for encouraging more research into the social intelligence of LLM Agents.

Figures (10)

• Figure 1: Sample evaluation subsets in CogMir framework. CogMir mirrors human cognitive bias and LLM Agents' systematic hallucination through social science experiments via representational social and cognitive phenomena. https://github.com/XuanL17/CogMir
• Figure 2: CogMir Framework. The framework is structured around four essential objects: "humans," LLM Agents, data, and discriminators. These objects interact within the framework to facilitate Q&A and Multi-Human-LLM Agent (Multi-H-A) interactions to mirror social science experimental settings and evaluations. CogMir features two communication modes and five Multi-H-A interaction combinations, enabling varied configurations to suit diverse social experimental needs. CogMir offers mirror cognitive bias samples (Fig. \ref{['fig:overview']}) and dynamic use cases open for expansion.
• Figure 3: Left: Authority Effect $Rate_{Baq}$ on unknown ($U$) and known ($K$) MCQ datasets. Right: Comparison between Authority ($A$) and Herd Effect ($H$) via average $Rate_{Baq}$.
• Figure 4: Rumor Chain Effect Visualization of semantic similarity ($SimCSE$-$RoBERTa_{large}$Simcse) via 15 LLM Agents Muti-A (Point-to-Point) scenario. S0 $\sim$ S9 denotes 10 stories.
• Figure 5: Radar plots for GPT models.