Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements

Tianjie Ju, Bowen Wang, Hao Fei, Mong-Li Lee, Wynne Hsu, Yun Li, Qianren Wang, Pengzhou Cheng, Zongru Wu, Haodong Zhao, Zhuosheng Zhang, Gongshen Liu

TL;DR

The paper investigates how disagreements affect robustness in LLM-based multi-agent systems by comparing single-path reasoning tasks to multi-path programming tasks. It shows that general disagreements broaden the search space and can boost performance, while task-critical disagreements undermine single-path reasoning but are more tolerable in programming where alternative solutions exist. Self-repair emerges when multiple solution paths exist, allowing MAS to bypass edited knowledge; however, excessive task-critical edits eventually degrade performance. The authors provide a formal, path-aware framework, comprehensive experiments across CounterFact, MQuAKE-cf, HumanEval, and GAIA, and actionable design guidelines to cultivate diversity and redundancy in MAS. Their findings offer practical implications for building robust, collaborative AI systems that leverage disagreement as a resource rather than a risk.

Abstract

Recent advances in Large Language Models (LLMs) have upgraded them from sophisticated text generators to autonomous agents capable of cooperation and tool use in multi-agent systems (MAS). However, it remains unclear how disagreements shape collective decision-making. In this paper, we revisit the role of disagreement and argue that general, partially overlapping disagreements prevent premature consensus and expand the explored solution space, while disagreements on task-critical steps can derail collaboration depending on the topology of solution paths. We investigate two collaborative settings with distinct path structures: collaborative reasoning (CounterFact, MQuAKE-cf), which typically follows a single evidential chain, whereas collaborative programming (HumanEval, GAIA) often adopts multiple valid implementations. Disagreements are instantiated as general heterogeneity among agents and as task-critical counterfactual knowledge edits injected into context or parameters. Experiments reveal that general disagreements consistently improve success by encouraging complementary exploration. By contrast, task-critical disagreements substantially reduce success on single-path reasoning, yet have a limited impact on programming, where agents can choose alternative solutions. Trace analyses show that MAS frequently bypasses the edited facts in programming but rarely does so in reasoning, revealing an emergent self-repair capability that depends on solution-path rather than scale alone. Our code is available at https://github.com/wbw625/MultiAgentRobustness.

When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements

TL;DR

The paper investigates how disagreements affect robustness in LLM-based multi-agent systems by comparing single-path reasoning tasks to multi-path programming tasks. It shows that general disagreements broaden the search space and can boost performance, while task-critical disagreements undermine single-path reasoning but are more tolerable in programming where alternative solutions exist. Self-repair emerges when multiple solution paths exist, allowing MAS to bypass edited knowledge; however, excessive task-critical edits eventually degrade performance. The authors provide a formal, path-aware framework, comprehensive experiments across CounterFact, MQuAKE-cf, HumanEval, and GAIA, and actionable design guidelines to cultivate diversity and redundancy in MAS. Their findings offer practical implications for building robust, collaborative AI systems that leverage disagreement as a resource rather than a risk.

Abstract

Recent advances in Large Language Models (LLMs) have upgraded them from sophisticated text generators to autonomous agents capable of cooperation and tool use in multi-agent systems (MAS). However, it remains unclear how disagreements shape collective decision-making. In this paper, we revisit the role of disagreement and argue that general, partially overlapping disagreements prevent premature consensus and expand the explored solution space, while disagreements on task-critical steps can derail collaboration depending on the topology of solution paths. We investigate two collaborative settings with distinct path structures: collaborative reasoning (CounterFact, MQuAKE-cf), which typically follows a single evidential chain, whereas collaborative programming (HumanEval, GAIA) often adopts multiple valid implementations. Disagreements are instantiated as general heterogeneity among agents and as task-critical counterfactual knowledge edits injected into context or parameters. Experiments reveal that general disagreements consistently improve success by encouraging complementary exploration. By contrast, task-critical disagreements substantially reduce success on single-path reasoning, yet have a limited impact on programming, where agents can choose alternative solutions. Trace analyses show that MAS frequently bypasses the edited facts in programming but rarely does so in reasoning, revealing an emergent self-repair capability that depends on solution-path rather than scale alone. Our code is available at https://github.com/wbw625/MultiAgentRobustness.

Paper Structure

This paper contains 39 sections, 4 equations, 3 figures, 21 tables.

Table of Contents

  1. Introduction
  2. Rethinking Multi-Agent Collaboration with Disagreements
  3. Information Flow in MAS
  4. Role of Disagreements in Task Completion
  5. From Fragility to Self-Repair
  6. From Fragility to Self-Repair
  7. Experiments
  8. Setup
  9. Evaluation Scenarios
  10. Collaborative Reasoning
  11. Collaborative Programming
  12. LLMs
  13. How Disagreements Affect Multi-Agent Decision-Making?
  14. Impact of General Disagreements
  15. Impact of Task-Critical Disagreements
  16. ...and 24 more sections

Figures (3)

  • Figure 1: LLM-based multi-agent collaboration under disagreements across single-chain evidence tasks (left) and tasks with multiple feasible solution paths (right). Insight i@: Partial disagreements expand the joint decision space of multi-agents. Insight ii@: unique-path tasks are brittle to local task-critical disagreements, whereas multi-path tasks can route around localized disagreements and still satisfy the task specification.
  • Figure 2: Two collaborative multi-agent settings used in our experiments. Left: Collaborative reasoning, where three agents jointly answer a fact-based question after multi-turn deliberation. Right: Collaborative programming, where one project manager, three coders, and one executor collaborate on implementation.
  • Figure 3: Impact of task-critical disagreement numbers on LLaMA-based HumanEval collaborative programming.