Conformity and Social Impact on AI Agents

TL;DR

The paper addresses how conformity manifests in AI agent collectives under social pressure by adapting Asch-style visual tasks to large multimodal LLMs and manipulating parameters from Social Impact Theory. It uses three tasks (line judgment, color recognition, dots estimation) and measures $p_{\text{wrong}}(N)$ and its $\mathrm{AUC}$ under varying group size $N$, source strength $S$, and immediacy $I$, including normative vs informational conditions. The findings show a robust conformity bias across models and tasks, with stronger effects for higher task difficulty, public visibility, stronger sources, and higher immediacy, while higher standalone performance does not consistently reduce susceptibility, especially near competence boundaries. These results reveal security vulnerabilities in AI-agent collectives, such as manipulation and misinformation opportunities, and underscore the need for safeguards, transparent consensus mechanisms, and protocols to preserve decision integrity in multi-agent deployments.

Abstract

As AI agents increasingly operate in multi-agent environments, understanding their collective behavior becomes critical for predicting the dynamics of artificial societies. This study examines conformity, the tendency to align with group opinions under social pressure, in large multimodal language models functioning as AI agents. By adapting classic visual experiments from social psychology, we investigate how AI agents respond to group influence as social actors. Our experiments reveal that AI agents exhibit a systematic conformity bias, aligned with Social Impact Theory, showing sensitivity to group size, unanimity, task difficulty, and source characteristics. Critically, AI agents achieving near-perfect performance in isolation become highly susceptible to manipulation through social influence. This vulnerability persists across model scales: while larger models show reduced conformity on simple tasks due to improved capabilities, they remain vulnerable when operating at their competence boundary. These findings reveal fundamental security vulnerabilities in AI agent decision-making that could enable malicious manipulation, misinformation campaigns, and bias propagation in multi-agent systems, highlighting the urgent need for safeguards in collective AI deployments.

Figures (12)

• Figure 1: Summary of the experimental setup. Illustration of the general experimental setting for the three visual tasks: line judgment, color recognition, and dots estimation. Each prompt consists of four main components. First, we present a single image representing the task. Second, we provide a textual description and the corresponding question. Third, we introduce social pressure by showing the model a sequence of prior responses (typically incorrect) sampled from a pool of naturalistic answers. In this section, we control the number of confederates $N$, i.e., the number of displayed replies. In the baseline condition ($N=0$), this section is omitted, simulating the absence of social influence. Finally, we append a closing instruction that asks the model to output only the label corresponding to its chosen answer, with no additional text. This restriction ensures that the model produces a single token, allowing us to extract and compare the generation logits between alternatives. Variants of this setup, used across the different experimental conditions, are described in Section \ref{['sec:methods']}. Full prompt examples are provided in Section \ref{['sec:SI_prompts']}.
• Figure 2: General drivers of conformity in LLMs.a) Probability of giving the wrong answer, used as a proxy for conformity, as a function of group size $N$ (i.e., number of confederates) for different models. Despite quantitative differences, all models exhibit a clear conformity bias, with some approaching near-complete alignment with the group ($p_{\text{wrong}} \simeq 1$). While certain models show a monotonic increase up to $N = 10$, others saturate around $N \sim 3$–$4$, consistent with the classic human pattern observed in Asch-type experiments. b) Conformity ($p_{\text{wrong}}$) as a function of $N$ for the same model (Qwen2.5 32B) across the three visual tasks (line judgment, color recognition, and dots estimation). The nearly overlapping curves indicate that the conformity effect is qualitatively stable across different task modalities. c) Conformity ($p_{\text{wrong}}$) as a function of $N$ for varying proportions of incorrect answers among the confederates. The percentage values indicate the fraction of wrong responses shown in the prompt (e.g., 50% corresponds to a random alternation between correct and incorrect replies). The strongest conformity effect occurs when all responses are wrong (100%), while it is almost completely suppressed when correct and incorrect answers are equally represented (50%). Importantly, even a small proportion of correct answers (e.g., 20%) is sufficient to markedly reduce conformity, showing that breaking unanimity strongly mitigates social influence. All results refer to the Qwen2.5 32B model on the color recognition task.
• Figure 3: Scaling of conformity with task difficulty, model performance and normative pressure.(a) Conformity, measured as the area under the $p_{\text{wrong}}(N)$ curve, as a function of task difficulty for the Qwen2.5 32B model in the color recognition task. Each point represents a set of images with a specific difficulty level, defined by the RGB distance between the reference and comparison colors. The harder the task, the higher the level of conformity, with a strong positive correlation (Spearman $\rho = 0.97$, $p < 10^{-10}$). (b) Conformity as a joint function of task difficulty and model performance (logit), for all models tested on the color recognition task. For each model, conformity, difficulty and performance values are normalized between their respective minimum and maximum to allow comparison across models. Each hexagon corresponds to a sample of images with a given difficulty and average logit of the correct answer. Moving vertically (increasing difficulty) leads to higher conformity, while moving horizontally (changing performance at fixed difficulty) shows no clear effect. A multivariate regression confirms that conformity is significantly influenced by task difficulty ($\beta = 0.657$, $p < 10^{-9}$), whereas performance (logit) has no significant impact ($p = 0.46$). Full regression results are reported in Figure \ref{['fig:figSI4']}. (c) Difference in conformity between public and private response conditions across all models. Each bar shows the change in conformity (AUC of $p_{\text{wrong}}(N)$) between the two settings. Positive values indicate higher conformity when responses are public, consistent with a normative effect. The $z$-scores, computed from a one-sample one-tailed $t$-test, quantify statistical significance of the difference and show that most models display a clear and significant increase in conformity, while only a few show negligible or no effect.
• Figure 4: The Social Impact Theory in AI agents.(a) Conformity ($p_{\text{wrong}}$) as a function of group size $N$ under different source strengths. The base condition refers to generic participants. When the confederates are described as authoritative figures (scientists, policemen, judges) conformity increases, while it decreases for low-authority sources such as kids or chatbots. (b) Conformity curves for different levels of social proximity in the Qwen2.5 32B model on the color recognition task. The model is assigned a nationality or a group identity, and confederate responses are labeled as coming from the same or a different group. Conformity rises when identity is shared and drops when it differs. The black curve shows the neutral baseline with no identity specified. (c) Average conformity, measured as the area under the curves in panel (a), shown separately for the three tasks as a function of source strength. A consistent positive trend emerges, with stronger sources producing up to a fifteen–twenty percent increase in conformity on average. Despite variability across models in the magnitude of the effect, the overall pattern remains stable, indicating that source strength systematically enhances conformity across different visual tasks. (d) Summary of the social proximity effect. The difference in conformity (AUC) between same–group and different–group conditions, averaged over nationality, ethnicity and generic group, is shown for representative models. The effect is large and consistent across tasks, reaching increases of nearly sixty percent in some cases. Complete results and their corresponding significance levels are reported in Section \ref{['sec:SI_general_results']}.
• Figure S1: Conformity as a function of group size across models for the line judgment task. Each panel shows the probability of giving the wrong answer, $p_{\text{wrong}}$, as a function of the number of confederates providing incorrect responses, $N$. (a) Qwen models; (b) Gemma models; (c) Ovis models; (d) Mistral model. Despite quantitative differences, all models exhibit a clear conformity effect, with $p_{\text{wrong}}$ increasing with $N$, indicating that social influence generalizes across architectures.