Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?

Hyeong Kyu Choi, Xiaojin Zhu, Sharon Li

TL;DR

The paper dissects Multi-Agent Debate (MAD) into majority voting and inter-agent debate, revealing that majority voting accounts for most gains across seven NLP benchmarks. It constructs a Dirichlet-Compound-Multinomial (DCM) Bayesian framework and proves that debate rounds induce a martingale in agents’ belief about the correct answer, so expected accuracy remains the same without targeted bias. The authors show that simple, theory-informed interventions that bias belief updates toward correct signals can improve MAD performance, suggesting ensembling methods are often more reliable and cost-effective than complex MAD designs. These findings provide practical guidance for designing MAD systems and highlight the value of preserving correct signals during inter-agent discussion, with implications for both closed- and open-ended tasks. The accompanying code release supports reproducibility and further exploration of these insights.

Abstract

Multi-Agent Debate~(MAD) has emerged as a promising paradigm for improving the performance of large language models through collaborative reasoning. Despite recent advances, the key factors driving MAD's effectiveness remain unclear. In this work, we disentangle MAD into two key components--Majority Voting and inter-agent Debate--and assess their respective contributions. Through extensive experiments across seven NLP benchmarks, we find that Majority Voting alone accounts for most of the performance gains typically attributed to MAD. To explain this, we propose a theoretical framework that models debate as a stochastic process. We prove that it induces a martingale over agents' belief trajectories, implying that debate alone does not improve expected correctness. Guided by these insights, we demonstrate that targeted interventions, by biasing the belief update toward correction, can meaningfully enhance debate effectiveness. Overall, our findings suggest that while MAD has potential, simple ensembling methods remain strong and more reliable alternatives in many practical settings. Code is released in https://github.com/deeplearning-wisc/debate-or-vote.

Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?

TL;DR

The paper dissects Multi-Agent Debate (MAD) into majority voting and inter-agent debate, revealing that majority voting accounts for most gains across seven NLP benchmarks. It constructs a Dirichlet-Compound-Multinomial (DCM) Bayesian framework and proves that debate rounds induce a martingale in agents’ belief about the correct answer, so expected accuracy remains the same without targeted bias. The authors show that simple, theory-informed interventions that bias belief updates toward correct signals can improve MAD performance, suggesting ensembling methods are often more reliable and cost-effective than complex MAD designs. These findings provide practical guidance for designing MAD systems and highlight the value of preserving correct signals during inter-agent discussion, with implications for both closed- and open-ended tasks. The accompanying code release supports reproducibility and further exploration of these insights.

Abstract

Multi-Agent Debate~(MAD) has emerged as a promising paradigm for improving the performance of large language models through collaborative reasoning. Despite recent advances, the key factors driving MAD's effectiveness remain unclear. In this work, we disentangle MAD into two key components--Majority Voting and inter-agent Debate--and assess their respective contributions. Through extensive experiments across seven NLP benchmarks, we find that Majority Voting alone accounts for most of the performance gains typically attributed to MAD. To explain this, we propose a theoretical framework that models debate as a stochastic process. We prove that it induces a martingale over agents' belief trajectories, implying that debate alone does not improve expected correctness. Guided by these insights, we demonstrate that targeted interventions, by biasing the belief update toward correction, can meaningfully enhance debate effectiveness. Overall, our findings suggest that while MAD has potential, simple ensembling methods remain strong and more reliable alternatives in many practical settings. Code is released in https://github.com/deeplearning-wisc/debate-or-vote.

Paper Structure

This paper contains 56 sections, 63 equations, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Multi-Agent Debate
  4. Debate vs. Voting: Formalization.
  5. Is Debate Really Necessary? A Closer Look at Debate vs. Voting
  6. Experimental Setup
  7. Baselines.
  8. Benchmarks.
  9. Key Observations
  10. Majority voting is surprisingly strong.
  11. Theoretical Analysis
  12. Definition 1. (Agent Response Generation via DCM)
  13. Theorem 1. (Majority Voting Success Probability)
  14. Remark 1.
  15. Definition 2. (Bayesian Belief Update from Neighbor Responses)
  16. ...and 41 more sections

Figures (4)

  • Figure 1: Majority Voting vs. MAD overview.
  • Figure 2: Majority Voting is the main contributor to MAD.
  • Figure 3: Accuracy improves with more agents.
  • Figure 4: Martingale process of the mean agent accuracy across debate rounds.