Can LLM Agents Really Debate? A Controlled Study of Multi-Agent Debate in Logical Reasoning

TL;DR

The study probes whether multi-agent debate (MAD) produces genuine deliberative reasoning in LLMs, using Knight–Knave–Spy puzzles with ground-truth evaluation. It combines a controlled experimental design (six design factors, diverse agent teams) with both outcome and process metrics to separate reasoning strength from coordination effects. Key findings show that intrinsic model strength and team diversity dominate debate success, while structural adjustments yield limited gains; successful debates are marked by inclusive, rationale-driven exchanges and the ability to overturn incorrect consensus. These insights inform the design of interpretable, truth-seeking MAD systems and highlight conditions under which debate improves or fails to improve logical reasoning.

Abstract

Multi-agent debate (MAD) has recently emerged as a promising framework for improving the reasoning performance of large language models (LLMs). Yet, whether LLM agents can genuinely engage in deliberative reasoning, beyond simple ensembling or majority voting, remains unclear. We address this question through a controlled study using the Knight--Knave--Spy logic puzzle, which enables precise, step-wise evaluation of debate outcomes and processes under verifiable ground truth. We systematically set up six structural and cognitive factors, including agent team size, composition, confidence visibility, debate order, debate depth, and task difficulty, to disentangle their respective effects on collective reasoning. Our results show that intrinsic reasoning strength and group diversity are the dominant drivers of debate success, while structural parameters such as order or confidence visibility offer limited gains. Beyond outcomes, process-level analyses identify key behavioral patterns: majority pressure suppresses independent correction, effective teams overturn incorrect consensus, and rational, validity-aligned reasoning most strongly predicts improvement. These findings provide valuable insights into how and why LLM debates succeed or fail, offering guidance for designing interpretable and truth-seeking multi-agent reasoning systems.

Figures (6)

• Figure 1: The illustration of the overall MAD framework design.
• Figure 2: Single-agent accuracy versus self-reported confidence over 100 medium-difficulty games (size=6). This benchmark guides the agent team formation by categorizing models into high / medium / low performance and confidence.
• Figure 3: Overall accuracy across controlled debate settings relative to the default anchor (A). Each panel isolates one factor while keeping others fixed. The Task difficulty factor, reflected by varying game sizes, is naturally incorporated in all subfigures.
• Figure 4: Heatmap on initial states. x-axis: initial stats. Here Ma = majority, Mi = Minorty, first C = chaos, second C = correct, and W = wrong. So MaC means the model starts with a majority and correct initial position in a debate round. y-axis: different models. This figure counts for each agent their initial position before each debate round, and if their final position after the debate round is correct or not (the correction rate).
• Figure 5: Correlations between states transition and final accuracy of the game instance. x-axis: 12 state transition possibilities. Left y-axis: counts of state transitions. Right y-axis, the weight each state transition contributes to the final accuracy of a game instance: for each game instance we calculate the percentage of players having correct role deduction, not a single 0/1 accuracy for the whole answer. We then employ linear regression methods to simulates the weights and present them in the figure. Higher weights means a state transition contributes positively to the final (smooth) accuracy.