Conversational Agents as Catalysts for Critical Thinking: Challenging Social Influence in Group Decision-making

TL;DR

The study tackles minority suppression and groupthink in power-imbalanced group decisions by testing an LLM-powered devil's advocate that provides counterarguments after every eight messages. Using 12 four-person Korean groups (three seniors and one junior) in a within-subject design, the AI intervention increased satisfaction with both process and outcomes, particularly for minority members, while psychological safety and cognitive workload showed non-significant changes or modest increases. The work offers empirical evidence that AI agents can shape group dynamics indirectly by fostering open dialogue and inclusive expression, and it proposes design guidelines for timing, specificity of counterarguments, and adaptability to group dynamics. This research has practical implications for deploying AI-assisted decision-support tools in organizational settings to mitigate conformity pressures without disrupting group cohesion.

Abstract

Group decision-making processes frequently suffer when social influence and power dynamics suppress minority viewpoints, leading to compliance and groupthink. Conversational agents can counteract these harmful dynamics by encouraging critical thinking. This study investigates how LLM-powered devil's advocate systems affect psychological safety, opinion expression, and satisfaction in power-imbalanced group dynamics. We conducted an experiment with 48 participants in 12 four-person groups, each containing three high-power (senior) and one low-power (junior) member. Each group completed decision tasks in both baseline and AI intervention conditions. Results show AI counterarguments fostered a more flexible atmosphere and significantly enhanced both process and outcome satisfaction for all participants, with particularly notable improvements for minority members. Cognitive workload increased slightly, though not significantly. This research contributes empirical evidence on how AI systems can effectively navigate power hierarchies to foster more inclusive decision-making environments, highlighting the importance of balancing intervention frequency, maintaining conversational flow, and preserving group cohesion.

Figures (7)

• Figure 1: System Overview: Our system architecture shows the interaction flow between the chat interface, database, and server components. The system processes both direct messages (DMs) and public chat through four main agents: (A) Summary Agent for public opinion analysis, (B) Conversation Agent for generating contextual counterarguments, and (C) AI Duplicate Checker for ensuring message novelty through cosine-similarity comparison.
• Figure 2: Self-reported metrics across conditions for (A) psychological safety, (B) satisfaction of decision-making process, (C) quality of decision-making outcome, (D) cognitive workload (NASA-TLX), (E) Perception of AI
• Figure 3: Hypothetical Model of the Trade-off between Critical Thinking and Group Satisfaction in Group Decision-making: This model illustrates how an LLM-powered devil's advocate acting as a naysayer can influence critical thinking and group dynamics. In a virtuous cycle, moderate stimulation of critical thinking (a) enhances decision-making outcomes (b), increasing group satisfaction (c), motivating continued use of LLM-powered devil's advocate (e), and fostering more critical thinking (d), with minimal negative impact (f). Conversely, in a vicious cycle, excessive stimulation (g) leads to cognitive overload and negative group dynamics (l), decreasing group satisfaction (k), reducing motivation to use LLM-powered devil's advocate (j), lowering decision-making quality (h), and further diminishing satisfaction and motivation (i). This model is theoretical and has not been empirically validated.
• Figure 4: Team Leader Promotion Review Task Instruction for Seniors
• Figure 5: Team Leader Promotion Review Task Instruction for Junior