Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unmasking Conversational Bias in AI Multiagent Systems

Erica Coppolillo, Giuseppe Manco, Luca Maria Aiello

TL;DR

The paper addresses the problem of biases in AI multiagent systems, where language models interacting in social-like settings can exhibit emergent biases not seen in isolated prompts. It introduces a framework with Social Agents and Opinion Signal Agents to simulate chatroom debates on polarizing topics, and demonstrates that substantial stance shifts occur in echo chambers, especially toward liberal positions; these shifts often escape detection by existing questionnaire-based bias tests. Across eight topics and seven models, the authors show that one-shot prompts do not reliably reveal conversational biases, and that dyadic conversations produce rapid, topic-dependent opinion changes. The work highlights the need for context-aware bias detection and mitigation when deploying conversational AIs in social environments.

Abstract

Detecting biases in the outputs produced by generative models is essential to reduce the potential risks associated with their application in critical settings. However, the majority of existing methodologies for identifying biases in generated text consider the models in isolation and neglect their contextual applications. Specifically, the biases that may arise in multi-agent systems involving generative models remain under-researched. To address this gap, we present a framework designed to quantify biases within multi-agent systems of conversational Large Language Models (LLMs). Our approach involves simulating small echo chambers, where pairs of LLMs, initialized with aligned perspectives on a polarizing topic, engage in discussions. Contrary to expectations, we observe significant shifts in the stance expressed in the generated messages, particularly within echo chambers where all agents initially express conservative viewpoints, in line with the well-documented political bias of many LLMs toward liberal positions. Crucially, the bias observed in the echo-chamber experiment remains undetected by current state-of-the-art bias detection methods that rely on questionnaires. This highlights a critical need for the development of a more sophisticated toolkit for bias detection and mitigation for AI multi-agent systems. The code to perform the experiments is publicly available at https://anonymous.4open.science/r/LLMsConversationalBias-7725.

Unmasking Conversational Bias in AI Multiagent Systems

TL;DR

The paper addresses the problem of biases in AI multiagent systems, where language models interacting in social-like settings can exhibit emergent biases not seen in isolated prompts. It introduces a framework with Social Agents and Opinion Signal Agents to simulate chatroom debates on polarizing topics, and demonstrates that substantial stance shifts occur in echo chambers, especially toward liberal positions; these shifts often escape detection by existing questionnaire-based bias tests. Across eight topics and seven models, the authors show that one-shot prompts do not reliably reveal conversational biases, and that dyadic conversations produce rapid, topic-dependent opinion changes. The work highlights the need for context-aware bias detection and mitigation when deploying conversational AIs in social environments.

Abstract

Detecting biases in the outputs produced by generative models is essential to reduce the potential risks associated with their application in critical settings. However, the majority of existing methodologies for identifying biases in generated text consider the models in isolation and neglect their contextual applications. Specifically, the biases that may arise in multi-agent systems involving generative models remain under-researched. To address this gap, we present a framework designed to quantify biases within multi-agent systems of conversational Large Language Models (LLMs). Our approach involves simulating small echo chambers, where pairs of LLMs, initialized with aligned perspectives on a polarizing topic, engage in discussions. Contrary to expectations, we observe significant shifts in the stance expressed in the generated messages, particularly within echo chambers where all agents initially express conservative viewpoints, in line with the well-documented political bias of many LLMs toward liberal positions. Crucially, the bias observed in the echo-chamber experiment remains undetected by current state-of-the-art bias detection methods that rely on questionnaires. This highlights a critical need for the development of a more sophisticated toolkit for bias detection and mitigation for AI multi-agent systems. The code to perform the experiments is publicly available at https://anonymous.4open.science/r/LLMsConversationalBias-7725.
Paper Structure (11 sections, 5 figures, 1 table)

This paper contains 11 sections, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Framework
  4. Social Agents
  5. Opinion Signal Agents
  6. Experimental Setup
  7. Results
  8. Bias in one-shot prompting
  9. Dyadic Conversations
  10. Discussion
  11. Conclusions and Future Work

Figures (5)

  • Figure 1: Illustration of our conversational framework. The social agents, by taking turns, generate a message based on their system prompt and the current history. For each generated content, the opinion presence agent determines whether an opinion is present. If an opinion is detected, the opinion signal agent is queried to provide the corresponding stance.
  • Figure 2: Unwarranted opinion change of LLM agents via one-shot prompting. The score represents the average across 10 different trials.
  • Figure 3: Percentage of chatroom simulations where at least one unwarranted opinion change occurs.
  • Figure 4: Number of agents changing opinion. Each dot indicates a chat where the underlying agents change opinion.
  • Figure 5: Percentage of Conservative conversations exhibiting an unwarranted opinion change (Y-axis) by varying the conversation length $M$ (X-axis).