An Electoral Approach to Diversify LLM-based Multi-Agent Collective Decision-Making

TL;DR

The paper analyzes the lack of diversity in CDM methods used by current LLM-based multi-agent systems and reframes the problem through social choice theory. It introduces GEDI, an electoral CDM interface that supports multiple ordinal voting rules, and demonstrates, via MCQA benchmarks, that voting-based CDM can improve reasoning and robustness, even with as few as three agents. The findings suggest that diversifying CDM methods—beyond dictatorial or plain plurality—yields meaningful performance gains and resilience, informing future MAS designs and benchmarking approaches. Overall, the work advocates systematic evaluation of diverse voting schemes to better harness collaborative LLM capabilities in distributed decision-making tasks.

Abstract

Modern large language models (LLMs) have exhibited cooperative synergy on complex task-solving, and collective decision-making (CDM) is a pivotal component in LLM-based multi-agent collaboration frameworks. Our survey on 52 recent such systems uncovers a severe lack of diversity, with a heavy reliance on dictatorial and plurality voting for CDM. Through the lens of social choice theory, we scrutinize widely-adopted CDM methods and identify their limitations. To enrich current landscape of LLM-based CDM, we present GEDI, an electoral CDM module that incorporates various ordinal preferential voting mechanisms. Our empirical case study across three benchmarks shows that the integration of certain CDM methods can markedly improve the reasoning capabilities and robustness of some leading LLMs, all without requiring intricate system designs. Additionally, we find that some CDM mechanisms generate positive synergies even with as few as three agents. The voting-based methods also demonstrate robustness against single points of failure, as well as diversity in terms of hit-rate@k and subject-wise impacts.

Figures (11)

• Figure 1: Distribution of CDM methods in 52 LLM-based multi-agent collaboration systems, denoting a severe lack of diversity.
• Figure 2: Comparison among different LLM-based multi-agent CDM structures: utilitarian, dictatorial, plurality and our expansion. Agenda refers to assigned tasks or interactive environment. Blue and green arrows denote interaction between agents and preference communication to CDM systems respectively. Rather than generate a single decision, GEDI uniquely outputs ordinal rankings, providing more information on agents' collective preferences.
• Figure 3: Accuracy comparison of voting ensembles of different sizes built on the same backbone models. The Range results of glm-4-9b is excluded for insufficient profiles (see Appendix \ref{['appendix:stats']}).
• Figure 4: Accuracy impact of increasing number of unreliable agents built on gpt-3.5 and gpt-4.
• Figure 5: Hit-rate@$k$ comparison of different voting rules utilising ballots given by voting agents. Green lines are drawn to highlight similar hit-rate@1.