The Effect of Belief Boxes and Open-mindedness on Persuasion

TL;DR

The paper tackles how explicit belief representations (belief boxes) and open-minded prompting influence belief change and persuasiveness in multi-agent LLM debates. It introduces a formal belief-revision model with belief strengths and an open-mindedness parameter, and evaluates it through Aporia and MMLU-based debates across multiple LLMs, plus predictive modeling with small LMs and classical ML. Key contributions include a detailed analysis of belief-box effects on argumentation, a framework for topic-driven debate with belief evaluation, and empirical evidence that open-mindedness and belief strength modulate persuasion and resilience to peer pressure. The findings advance explainability and control in AI-driven reasoning systems and shed light on how group dynamics shape belief dynamics in automated discourse.

Abstract

As multi-agent systems are increasingly utilized for reasoning and decision-making applications, there is a greater need for LLM-based agents to have something resembling propositional beliefs. One simple method for doing so is to include statements describing beliefs maintained in the prompt space (in what we'll call their belief boxes). But when agents have such statements in belief boxes, how does it actually affect their behaviors and dispositions towards those beliefs? And does it significantly affect agents' ability to be persuasive in multi-agent scenarios? Likewise, if the agents are given instructions to be open-minded, how does that affect their behaviors? We explore these and related questions in a series of experiments. Our findings confirm that instructing agents to be open-minded affects how amenable they are to belief change. We show that incorporating belief statements and their strengths influences an agent's resistance to (and persuasiveness against) opposing viewpoints. Furthermore, it affects the likelihood of belief change, particularly when the agent is outnumbered in a debate by opposing viewpoints, i.e., peer pressure scenarios. The results demonstrate the feasibility and validity of the belief box technique in reasoning and decision-making tasks.

Figures (7)

• Figure 1: Belief change for different open-mindedness levels. "Misalign" and "align" refer to agents' starting arguments.
• Figure 2: Mean belief score of the target agent based on the persuading agent with different beliefs and belief scores.
• Figure 3: Our prompting approach for BFI-2 tests.
• Figure 4: Our prompting approach for belief change experiments.
• Figure 5: Our prompting approach for the persuading agent for belief box effects on persuasiveness.