Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MindScope: Exploring cognitive biases in large language models through Multi-Agent Systems

Zhentao Xie, Jiabao Zhao, Yilei Wang, Jinxin Shi, Yanhong Bai, Xingjiao Wu, Liang He

TL;DR

The 'MindScope' dataset is introduced, which distinctively integrates static and dynamic elements, and a multi-agent detection method applicable to a wide range of detection tasks, which integrates Retrieval-Augmented Generation, competitive debate, and a reinforcement learning-based decision module is introduced.

Abstract

Detecting cognitive biases in large language models (LLMs) is a fascinating task that aims to probe the existing cognitive biases within these models. Current methods for detecting cognitive biases in language models generally suffer from incomplete detection capabilities and a restricted range of detectable bias types. To address this issue, we introduced the 'MindScope' dataset, which distinctively integrates static and dynamic elements. The static component comprises 5,170 open-ended questions spanning 72 cognitive bias categories. The dynamic component leverages a rule-based, multi-agent communication framework to facilitate the generation of multi-round dialogues. This framework is flexible and readily adaptable for various psychological experiments involving LLMs. In addition, we introduce a multi-agent detection method applicable to a wide range of detection tasks, which integrates Retrieval-Augmented Generation (RAG), competitive debate, and a reinforcement learning-based decision module. Demonstrating substantial effectiveness, this method has shown to improve detection accuracy by as much as 35.10% compared to GPT-4. Codes and appendix are available at https://github.com/2279072142/MindScope.

MindScope: Exploring cognitive biases in large language models through Multi-Agent Systems

TL;DR

The 'MindScope' dataset is introduced, which distinctively integrates static and dynamic elements, and a multi-agent detection method applicable to a wide range of detection tasks, which integrates Retrieval-Augmented Generation, competitive debate, and a reinforcement learning-based decision module is introduced.

Abstract

Detecting cognitive biases in large language models (LLMs) is a fascinating task that aims to probe the existing cognitive biases within these models. Current methods for detecting cognitive biases in language models generally suffer from incomplete detection capabilities and a restricted range of detectable bias types. To address this issue, we introduced the 'MindScope' dataset, which distinctively integrates static and dynamic elements. The static component comprises 5,170 open-ended questions spanning 72 cognitive bias categories. The dynamic component leverages a rule-based, multi-agent communication framework to facilitate the generation of multi-round dialogues. This framework is flexible and readily adaptable for various psychological experiments involving LLMs. In addition, we introduce a multi-agent detection method applicable to a wide range of detection tasks, which integrates Retrieval-Augmented Generation (RAG), competitive debate, and a reinforcement learning-based decision module. Demonstrating substantial effectiveness, this method has shown to improve detection accuracy by as much as 35.10% compared to GPT-4. Codes and appendix are available at https://github.com/2279072142/MindScope.
Paper Structure (28 sections, 7 figures, 3 tables)

This paper contains 28 sections, 7 figures, 3 tables.

Table of Contents

  1. Related Work
  2. Cognitive biases in LLMs
  3. LLM-based Multi-Agent System
  4. Problem Definition
  5. Dataset Construction
  6. Static dataset construction
  7. Dynamic dataset construction
  8. Validation of the validity of assessment tools
  9. Method
  10. The foundational architecture of RuleGen
  11. Rule-Based Multi-Agent Communication
  12. Automated rule-based scenario construction
  13. Multi-Dimensional Agent Behavior Monitoring
  14. Detecting Cognitive Bias Without Labels
  15. Cognitive Bias Recognition and Detection
  16. ...and 13 more sections

Figures (7)

  • Figure 1: Overview of the Construction of the MindScope Dataset.
  • Figure 2: RuleGen is a rule-based, multi-dimensional behavior monitoring multi-agent communication framework that enables users to automate scenario construction through no-code operations. It offers researchers an efficient tool for studying large-scale model scenario simulations.
  • Figure 3: Overview of learnable multi-agent detection method based on RAG, competitive debate and decision module.
  • Figure 4: Cognitive Bias Frequency in LLMs
  • Figure 5: Cognitive Bias Frequency in LLMs
  • ...and 2 more figures