Improving Multi-Agent Debate with Sparse Communication Topology

TL;DR

This work investigates sparse communication topologies within the multi-agent debate (MAD) framework, showing that neighbor-connected graphs with reduced connectivity can match or exceed fully-connected MAD in accuracy while dramatically cutting input-token costs. By quantifying topology via density $D=\frac{2|\mathcal{E}|}{|\mathcal{V}|(|\mathcal{V}|-1)}$ (with $D=1$ for fully-connected and $D=\frac{2}{|\mathcal{V}|-1}$ for neighbor-connected) and evaluating on text and multimodal reasoning as well as alignment labeling, the authors demonstrate robust performance gains and cost savings across GPT-3.5, GPT-4, and Mistral 7B. The study further shows that sparse MAD generalizes to multimodal inputs and that assigning stronger LLMs to highly central nodes improves overall results. These findings highlight the efficiency and effectiveness of sparse, topology-aware collaboration in large-scale agent societies and offer practical guidance for designing communication topologies in AI debate and feedback systems.

Abstract

Multi-agent debate has proven effective in improving large language models quality for reasoning and factuality tasks. While various role-playing strategies in multi-agent debates have been explored, in terms of the communication among agents, existing approaches adopt a brute force algorithm -- each agent can communicate with all other agents. In this paper, we systematically investigate the effect of communication connectivity in multi-agent systems. Our experiments on GPT and Mistral models reveal that multi-agent debates leveraging sparse communication topology can achieve comparable or superior performance while significantly reducing computational costs. Furthermore, we extend the multi-agent debate framework to multimodal reasoning and alignment labeling tasks, showcasing its broad applicability and effectiveness. Our findings underscore the importance of communication connectivity on enhancing the efficiency and effectiveness of the "society of minds" approach.

Figures (9)

• Figure 1: Accuracy (top) and inference input cost (middle) comparison of multi-agent debate system between fully-connected (bottom left) and neighbor-connected (bottom right) communication topologies.
• Figure 2: Communication topology of 6 agents with various sparsity. From left to right, the densities are 1 (fully-connected), $\frac{4}{5}$, $\frac{3}{5}$, and $\frac{2}{5}$ (neighbor-connected).
• Figure 3: Probability of a single agent generating correct answers given $n$ reference solutions, with $p$ representing the correctness of these solutions. Monte Carlo sampling was performed on three questions, each with 100 runs.
• Figure 4: Effective debate rounds for each topology design in reasoning and alignment labeling tasks.
• Figure 5: Isotropic communication topology with two setups: the stronger LLM has low centrality (left) and high centrality (right).