M3MAD-Bench: Are Multi-Agent Debates Really Effective Across Domains and Modalities?

TL;DR

The paper tackles fragmented MAD evaluations and unimodal limitations by introducing M3MAD-Bench, a unified benchmark spanning five domains and multiple modalities with multi-dimensional evaluation. It systematically analyzes collaborative and adversarial MAD methods across diverse base models, revealing that collaboration generally outperforms adversarial strategies and that gains are task- and modality-dependent. The study highlights that improvements are more pronounced on reasoning-intensive tasks and multimodal settings, while costs in tokens and latency can be substantial. A key finding is that collective delusion is a major failure mode, underscoring the need for robust aggregation and error-detection mechanisms; the benchmark provides a reproducible platform to guide future MAD research and development.

Abstract

As an agent-level reasoning and coordination paradigm, Multi-Agent Debate (MAD) orchestrates multiple agents through structured debate to improve answer quality and support complex reasoning. However, existing research on MAD suffers from two fundamental limitations: evaluations are conducted under fragmented and inconsistent settings, hindering fair comparison, and are largely restricted to single-modality scenarios that rely on textual inputs only. To address these gaps, we introduce M3MAD-Bench, a unified and extensible benchmark for evaluating MAD methods across Multi-domain tasks, Multi-modal inputs, and Multi-dimensional metrics. M3MAD-Bench establishes standardized protocols over five core task domains: Knowledge, Mathematics, Medicine, Natural Sciences, and Complex Reasoning, and systematically covers both pure text and vision-language datasets, enabling controlled cross-modality comparison. We evaluate MAD methods on nine base models spanning different architectures, scales, and modality capabilities. Beyond accuracy, M3MAD-Bench incorporates efficiency-oriented metrics such as token consumption and inference time, providing a holistic view of performance--cost trade-offs. Extensive experiments yield systematic insights into the effectiveness, robustness, and efficiency of MAD across text-only and multimodal scenarios. We believe M3MAD-Bench offers a reliable foundation for future research on standardized MAD evaluation. The code is available at http://github.com/liaolea/M3MAD-Bench.

Figures (8)

• Figure 1: Overview of M3MAD-Bench. The framework comprises three pillars: (1) Multi-Domain Coverage spanning diverse subjects and complex reasoning; (2) Multi-Modal Support integrating both text-only and vision-language datasets; and (3) Multi-Dimensional Evaluation assessing Collaborative and Adversarial debate strategies via accuracy and cost metrics.
• Figure 2: Accuracy vs. token consumption (#Tokens/Query) on log-scale. Results are shown for (a) LLaMA3.1-8B on unimodal datasets and (b) Qwen2.5-VL-7B on multimodal datasets. Left and right plots correspond to Input and Output tokens, respectively.
• Figure 3: Accuracy vs. Inference Time (GPT-4o-mini). Bars denote accuracy (left) and lines denote time cost (right) across (a) unimodal and (b) multimodal datasets.
• Figure 4: Performance across debate rounds. Evaluated on (a) MATH (LLaMA3.1-8B) and (b) MathVista (Qwen2.5-VL-7B).
• Figure 5: Case study of LLM Debate: (a) Misleading of Correct Answers and (b) Incorrect Answers Corrected.